[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 314761491 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.437876] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.440582] pid_max: default: 32768 minimum: 301 [ 0.442058] Security Framework initialized [ 0.443547] SELinux: Initializing. [ 0.444695] SELinux: Starting in permissive mode [ 0.446194] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.450262] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.453996] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.457218] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.460420] Initializing cgroup subsys memory [ 0.462050] Initializing cgroup subsys devices [ 0.463537] Initializing cgroup subsys freezer [ 0.465192] Initializing cgroup subsys net_cls [ 0.467129] Initializing cgroup subsys blkio [ 0.468583] Initializing cgroup subsys perf_event [ 0.470034] Initializing cgroup subsys hugetlb [ 0.471320] Initializing cgroup subsys pids [ 0.473989] Initializing cgroup subsys net_prio [ 0.475405] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.478340] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.479967] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.482194] tlb_flushall_shift: 6 [ 0.483275] FEATURE SPEC_CTRL Present [ 0.484358] FEATURE IBPB_SUPPORT Present [ 0.485415] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.487083] Spectre V2 : Vulnerable [ 0.488044] Speculative Store Bypass: Vulnerable [ 0.490320] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.498035] ACPI: Core revision 20130517 [ 0.500866] ACPI: All ACPI Tables successfully acquired [ 0.502687] ftrace: allocating 30294 entries in 119 pages [ 0.557479] Enabling x2apic [ 0.558412] Enabled x2apic [ 0.559584] Switched APIC routing to physical x2apic. [ 0.562730] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.564516] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.567357] TSC deadline timer enabled [ 0.567518] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.569982] ... version: 2 [ 0.571148] ... bit width: 48 [ 0.572283] ... generic registers: 4 [ 0.573420] ... value mask: 0000ffffffffffff [ 0.574882] ... max period: 00007fffffffffff [ 0.576194] ... fixed-purpose events: 3 [ 0.577182] ... event mask: 000000070000000f [ 0.578547] KVM setup paravirtual spinlock [ 0.583382] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.586601] KVM setup async PF for cpu 1 [ 0.588134] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.590806] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.593640] KVM setup async PF for cpu 2 [ 0.581671] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.594132] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.596444] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.598024] Brought up 4 CPUs [ 0.598040] KVM setup async PF for cpu 3 [ 0.598047] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.601291] smpboot: Max logical packages: 1 [ 0.602427] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.606374] devtmpfs: initialized [ 0.607560] x86/mm: Memory block size: 128MB [ 0.612284] EVM: security.selinux [ 0.613238] EVM: security.ima [ 0.613976] EVM: security.capability [ 0.617958] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.620986] NET: Registered protocol family 16 [ 0.622980] cpuidle: using governor haltpoll [ 0.625082] ACPI: bus type PCI registered [ 0.626593] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.629367] PCI: Using configuration type 1 for base access [ 0.631604] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.641687] ACPI: Added _OSI(Module Device) [ 0.643419] ACPI: Added _OSI(Processor Device) [ 0.644884] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.646393] ACPI: Added _OSI(Processor Aggregator Device) [ 0.648398] ACPI: Added _OSI(Linux-Dell-Video) [ 0.650753] ACPI: EC: Look up EC in DSDT [ 0.653251] ACPI: Interpreter enabled [ 0.654552] ACPI: (supports S0 S3 S4 S5) [ 0.656111] ACPI: Using IOAPIC for interrupt routing [ 0.657704] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.661281] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.668517] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.670775] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.673546] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.675664] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.679512] acpiphp: Slot [2] registered [ 0.680812] acpiphp: Slot [3] registered [ 0.682137] acpiphp: Slot [4] registered [ 0.683418] acpiphp: Slot [5] registered [ 0.684847] acpiphp: Slot [6] registered [ 0.685994] acpiphp: Slot [7] registered [ 0.687196] acpiphp: Slot [8] registered [ 0.688474] acpiphp: Slot [9] registered [ 0.689745] acpiphp: Slot [10] registered [ 0.691270] acpiphp: Slot [11] registered [ 0.692634] acpiphp: Slot [12] registered [ 0.694145] acpiphp: Slot [13] registered [ 0.695685] acpiphp: Slot [14] registered [ 0.697180] acpiphp: Slot [15] registered [ 0.698624] acpiphp: Slot [16] registered [ 0.700126] acpiphp: Slot [17] registered [ 0.701550] acpiphp: Slot [18] registered [ 0.703337] acpiphp: Slot [19] registered [ 0.704897] acpiphp: Slot [20] registered [ 0.706320] acpiphp: Slot [21] registered [ 0.707879] acpiphp: Slot [22] registered [ 0.709430] acpiphp: Slot [23] registered [ 0.710983] acpiphp: Slot [24] registered [ 0.712469] acpiphp: Slot [25] registered [ 0.713930] acpiphp: Slot [26] registered [ 0.715333] acpiphp: Slot [27] registered [ 0.716898] acpiphp: Slot [28] registered [ 0.718007] acpiphp: Slot [29] registered [ 0.719032] acpiphp: Slot [30] registered [ 0.720077] acpiphp: Slot [31] registered [ 0.721083] PCI host bridge to bus 0000:00 [ 0.722020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.723591] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.725477] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.727716] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.730352] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.732997] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.734647] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.735456] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.736421] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.742158] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.744631] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.747512] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.749428] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.751501] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.753878] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.754498] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.756674] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.759256] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.761131] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.770191] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.771791] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.772996] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.777800] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.782536] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.803094] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.809649] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.813927] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.817647] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.836166] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.840804] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.843466] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.847078] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.855797] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.861272] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.863836] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.866323] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.877967] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.882589] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.886163] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.888416] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.899042] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.902489] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.905123] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.907537] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.920375] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.924418] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.927369] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.929667] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.931851] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.934047] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.937779] vgaarb: loaded [ 0.938551] SCSI subsystem initialized [ 0.939730] ACPI: bus type USB registered [ 0.941081] usbcore: registered new interface driver usbfs [ 0.942894] usbcore: registered new interface driver hub [ 0.944614] usbcore: registered new device driver usb [ 0.946434] PCI: Using ACPI for IRQ routing [ 0.947502] PCI: pci_cache_line_size set to 64 bytes [ 0.947772] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.947777] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.947779] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.948827] NetLabel: Initializing [ 0.950064] NetLabel: domain hash size = 128 [ 0.951611] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.952959] NetLabel: unlabeled traffic allowed by default [ 0.955019] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.956898] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.962439] amd_nb: Cannot enumerate AMD northbridges [ 0.964468] Switched to clocksource kvm-clock [ 0.984144] pnp: PnP ACPI init [ 0.985348] ACPI: bus type PNP registered [ 0.986802] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.986898] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.986951] pnp 00:02: [dma 2] [ 0.986986] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.987072] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.987155] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.987222] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.987531] pnp: PnP ACPI: found 6 devices [ 0.988592] ACPI: bus type PNP unregistered [ 1.000479] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.000485] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.000488] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.000491] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.000493] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.000564] NET: Registered protocol family 2 [ 1.002369] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.005100] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.007877] TCP: Hash tables configured (established 32768 bind 32768) [ 1.009915] TCP: reno registered [ 1.011276] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.013654] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.016187] NET: Registered protocol family 1 [ 1.018834] RPC: Registered named UNIX socket transport module. [ 1.020766] RPC: Registered udp transport module. [ 1.022649] RPC: Registered tcp transport module. [ 1.024664] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.026474] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.028511] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.030857] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.033440] PCI: CLS 0 bytes, default 64 [ 1.034225] Unpacking initramfs... [ 2.382184] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.385784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.387891] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.391234] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.394156] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.395976] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.397856] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.402376] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 2.403780] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.403831] Initialise system trusted keyring [ 2.440518] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.442712] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.449521] zpool: loaded [ 2.450444] zbud: loaded [ 2.451619] VFS: Disk quotas dquot_6.6.0 [ 2.453209] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.456138] NFS: Registering the id_resolver key type [ 2.457625] Key type id_resolver registered [ 2.458851] Key type id_legacy registered [ 2.460002] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.462365] Key type big_key registered [ 2.463761] SELinux: Registering netfilter hooks [ 2.465950] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 2.468362] cryptomgr_test (61) used greatest stack depth: 13968 bytes left [ 2.471126] NET: Registered protocol family 38 [ 2.471137] Key type asymmetric registered [ 2.471140] Asymmetric key parser 'x509' registered [ 2.471271] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.471441] io scheduler noop registered [ 2.471446] io scheduler deadline registered (default) [ 2.471543] io scheduler cfq registered [ 2.471549] io scheduler mq-deadline registered [ 2.471553] io scheduler kyber registered [ 2.474031] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.474041] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.489368] intel_idle: does not run on family 6 model 62 [ 2.489545] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.491763] ACPI: Power Button [PWRF] [ 2.493491] GHES: HEST is not enabled! [ 2.550523] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.612740] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.733559] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.797015] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.924528] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.963083] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.993027] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.996561] Non-volatile memory driver v1.3 [ 2.998688] Linux agpgart interface v0.103 [ 3.000258] crash memory driver: version 1.1 [ 3.002453] nbd: registered device at major 43 [ 3.008899] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.008954] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.009000] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.009040] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.009084] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.016194] virtio_blk virtio1: [vda] 115440 512-byte logical blocks (59.1 MB/56.3 MiB) [ 3.022419] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.022486] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.022521] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.022551] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.022580] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.031091] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.040143] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.040187] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.040218] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.040247] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.040276] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.046320] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.053760] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.053810] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.053852] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.053889] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.053934] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.061289] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.068866] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.068910] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.068941] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.068970] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.069004] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.075085] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.079768] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.079817] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.079867] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.079917] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.079958] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.086029] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.090907] rdac: device handler registered [ 3.092541] hp_sw: device handler registered [ 3.094069] emc: device handler registered [ 3.095574] libphy: Fixed MDIO Bus: probed [ 3.103584] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.106019] ehci-pci: EHCI PCI platform driver [ 3.107558] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.109606] ohci-pci: OHCI PCI platform driver [ 3.110774] uhci_hcd: USB Universal Host Controller Interface driver [ 3.113215] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.116693] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.118548] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.121042] mousedev: PS/2 mouse device common for all mice [ 3.123990] rtc_cmos 00:05: RTC can wake from S4 [ 3.126049] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.128902] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.131859] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.136013] hidraw: raw HID events driver (C) Jiri Kosina [ 3.138690] usbcore: registered new interface driver usbhid [ 3.142816] usbhid: USB HID core driver [ 3.144625] drop_monitor: Initializing network drop monitor service [ 3.147481] Netfilter messages via NETLINK v0.30. [ 3.149234] TCP: cubic registered [ 3.150160] Initializing XFRM netlink socket [ 3.152060] NET: Registered protocol family 10 [ 3.154435] NET: Registered protocol family 17 [ 3.155992] Key type dns_resolver registered [ 3.158243] mce: Using 10 MCE banks [ 3.159753] Loading compiled-in X.509 certificates [ 3.163813] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.166644] registered taskstats version 1 [ 3.170718] modprobe (72) used greatest stack depth: 12992 bytes left [ 3.176871] Key type trusted registered [ 3.183720] Key type encrypted registered [ 3.185717] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.189824] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.193440] rtc_cmos 00:05: setting system clock to 2025-05-01 19:52:59 UTC (1746129179) [ 3.197130] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.199728] Write protecting the kernel read-only data: 12288k [ 3.201978] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.204783] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.216603] random: systemd: uninitialized urandom read (16 bytes read) [ 3.220631] random: systemd: uninitialized urandom read (16 bytes read) [ 3.223256] random: systemd: uninitialized urandom read (16 bytes read) [ 3.227430] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.234329] systemd[1]: Detected virtualization kvm. [ 3.236179] systemd[1]: Detected architecture x86-64. [ 3.237863] systemd[1]: Running in initial RAM disk. [ 3.243159] systemd[1]: No hostname configured. [ 3.244924] systemd[1]: Set hostname to . [ 3.246916] random: systemd: uninitialized urandom read (16 bytes read) [ 3.249101] systemd[1]: Initializing machine ID from random generator. [ 3.299527] random: systemd: uninitialized urandom read (16 bytes read) [ 3.302006] random: systemd: uninitialized urandom read (16 bytes read) [ 3.304125] random: systemd: uninitialized urandom read (16 bytes read) [ 3.306417] random: systemd: uninitialized urandom read (16 bytes read) [ 3.309733] random: systemd: uninitialized urandom read (16 bytes read) [ 3.312158] random: systemd: uninitialized urandom read (16 bytes read) [ 3.322597] systemd[1]: Reached target Timers. [ 3.325719] systemd[1]: Reached target Swap. [ 3.329026] systemd[1]: Reached target Local File Systems. [ 3.334517] systemd[1]: Created slice Root Slice. [ 3.339027] systemd[1]: Created slice System Slice. [ 3.342641] systemd[1]: Listening on udev Control Socket. [ 3.346853] systemd[1]: Reached target Slices. [ 3.350844] systemd[1]: Listening on udev Kernel Socket. [ 3.356112] systemd[1]: Listening on Journal Socket. [ 3.360930] systemd[1]: Reached target Sockets. [ 3.367835] systemd[1]: Starting Load Kernel Modules... [ 3.372979] systemd[1]: Starting dracut cmdline hook... [ 3.378362] systemd[1]: Starting Journal Service... [ 3.383454] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.390979] systemd[1]: Starting Setup Virtual Console... [ 3.396426] systemd[1]: Started Load Kernel Modules. [ 3.401531] tsc: Refined TSC clocksource calibration: 2399.986 MHz [ 3.406400] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.412299] systemd[1]: Started Setup Virtual Console. [ 3.418425] systemd[1]: Starting Create Static Device Nodes in /dev... [ 3.424805] systemd[1]: Starting Apply Kernel Variables... [ 3.431608] systemd[1]: Started Create Static Device Nodes in /dev. [ 3.437776] systemd[1]: Started Apply Kernel Variables. [ 3.442522] systemd[1]: Started Journal Service. [ 3.445439] random: fast init done [ 3.957601] libata version 3.00 loaded. [ 3.959900] ata_piix 0000:00:01.1: version 2.13 [ 3.973571] scsi host0: ata_piix [ 3.980746] scsi host1: ata_piix [ 3.982444] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.986805] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 3.997939] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.279355] ip (343) used greatest stack depth: 12336 bytes left [ 10.568615] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 11.293847] systemd-journald[103]: Received SIGTERM from PID 1 (systemd). [ 11.712360] SELinux: Disabled at runtime. [ 11.714311] SELinux: Unregistering netfilter hooks [ 11.823926] ip_tables: (C) 2000-2006 Netfilter Core Team [ 11.830319] systemd[1]: Inserted module 'ip_tables' [ 13.088063] systemd-journald[569]: Received request to flush runtime journal from PID 1 [ 13.421254] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 13.539093] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.600937] cryptd: max_cpu_qlen set to 1000 [ 13.720450] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 13.766916] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 13.868201] AVX version of gcm_enc/dec engaged. [ 13.869503] AES CTR mode by8 optimization enabled [ 13.926121] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 13.937184] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 14.275680] EDAC MC: Ver: 3.0.0 [ 14.301012] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 14.301024] EDAC sbridge: Ver: 1.1.2 [ 19.190954] loadkeys (776) used greatest stack depth: 12304 bytes left [ 19.325989] mount.nfs (780) used greatest stack depth: 10704 bytes left [ 38.271046] device-mapper: uevent: version 1.0.3 [ 38.273261] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 46.224616] libcfs: loading out-of-tree module taints kernel. [ 46.227026] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 46.302272] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 53.036587] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 53.331977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.341211] alg: No test for adler32 (adler32-zlib) [ 54.319841] Lustre: Lustre: Build Version: 2.16.54_86_g77fe414 [ 54.576643] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 54.579682] LNet: Accept secure, port 988 [ 56.154371] Key type lgssc registered [ 56.682737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.380699] icp: module license 'CDDL' taints kernel. [ 61.382720] Disabling lock debugging due to kernel taint [ 63.464275] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 67.471507] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.328543] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.658518] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 77.665591] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.280342] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 81.289403] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.549309] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 91.803421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.838376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.855093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.022426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 93.056024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.097242] Lustre: lustre-MDT0000: new disk, initializing [ 93.165431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.178913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 93.249992] mount.lustre (7219) used greatest stack depth: 10208 bytes left [ 94.710839] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.462103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.513299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 101.564510] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 101.580151] Lustre: 8376:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 101.595585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 101.599599] Lustre: Skipped 1 previous similar message [ 101.647003] Lustre: lustre-MDT0001: new disk, initializing [ 101.697921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.732672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 101.746096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 103.424554] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.658351] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 111.691435] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 111.699821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.745041] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 111.752768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.859376] Lustre: 9578:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 111.876619] Lustre: lustre-OST0000: new disk, initializing [ 111.881191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 111.918946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 114.407165] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.726154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 120.739188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 120.863488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 122.735853] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 122.746400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.786188] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 122.793599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 122.843573] Lustre: 10918:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 122.870089] Lustre: lustre-OST0001: new disk, initializing [ 122.876825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 122.920315] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 126.007255] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.068534] random: crng init done [ 130.674907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 130.679056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 130.718912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 133.213104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.679441] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 145.037119] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing check_logdir /tmp/testlogs/ [ 146.627598] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing yml_node [ 148.738253] Lustre: DEBUG MARKER: Client: 2.16.54.86 [ 149.951223] Lustre: DEBUG MARKER: MDS: 2.16.54.86 [ 151.004315] Lustre: DEBUG MARKER: OSS: 2.16.54.86 [ 151.804498] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu May 1 15:55:27 EDT 2025 [ 159.730418] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 160.421245] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 161.233630] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 851 411b [ 162.089413] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 162.918349] Lustre: DEBUG MARKER: === sanity: start setup 15:55:38 (1746129338) === [ 164.654607] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing check_config_client /mnt/lustre [ 174.850738] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 176.710109] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 178.002838] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.732794] Lustre: DEBUG MARKER: === sanity: finish setup 15:55:56 (1746129356) === [ 184.265196] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 15:55:59 (1746129359) [ 185.971918] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 186.957936] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 15:56:02 (1746129362) [ 190.995798] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 15:56:06 (1746129366) [ 278.643834] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 15:57:34 (1746129454) [ 281.056507] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 15:57:36 (1746129456) [ 281.566027] Lustre: *** cfs_fail_loc=15b, val=0*** [ 281.568701] Lustre: *** cfs_fail_loc=15b, val=0*** [ 284.234532] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 15:57:40 (1746129460) [ 287.247397] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 15:57:43 (1746129463) [ 287.704431] Lustre: *** cfs_fail_loc=19a, val=0*** [ 288.454905] Lustre: *** cfs_fail_loc=19a, val=0*** [ 288.458010] Lustre: Skipped 1 previous similar message [ 289.649021] Lustre: *** cfs_fail_loc=19a, val=0*** [ 289.650601] Lustre: Skipped 2 previous similar messages [ 291.949853] Lustre: *** cfs_fail_loc=19a, val=0*** [ 291.951574] Lustre: Skipped 5 previous similar messages [ 295.989991] Lustre: *** cfs_fail_loc=19a, val=0*** [ 295.992320] Lustre: Skipped 9 previous similar messages [ 304.253347] Lustre: *** cfs_fail_loc=19a, val=0*** [ 304.255187] Lustre: Skipped 21 previous similar messages [ 320.282693] Lustre: *** cfs_fail_loc=19a, val=0*** [ 320.286228] Lustre: Skipped 35 previous similar messages [ 333.921634] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 15:58:29 (1746129509) [ 334.338689] Lustre: *** cfs_fail_loc=188, val=0*** [ 335.541019] Lustre: *** cfs_fail_loc=189, val=0*** [ 338.744041] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 339.370200] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 15:58:35 (1746129515) [ 341.506887] Lustre: lustre-MDD0000: changelog on [ 342.549426] Lustre: lustre-MDD0001: changelog on [ 346.083229] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format [ 347.157876] Lustre: lustre-MDD0001: changelog off [ 348.241864] Lustre: lustre-MDD0000: changelog off [ 349.360191] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 15:58:45 (1746129525) [ 352.786079] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 15:58:48 (1746129528) [ 355.699615] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 15:58:51 (1746129531) [ 419.252711] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 15:59:55 (1746129595) [ 425.334024] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 16:00:01 (1746129601) [ 428.240867] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 428.827347] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 16:00:04 (1746129604) [ 431.872303] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 16:00:07 (1746129607) [ 453.459287] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 16:00:29 (1746129629) [ 454.730034] Lustre: *** cfs_fail_loc=725, val=0*** [ 458.340374] Lustre: *** cfs_fail_loc=725, val=0*** [ 461.881495] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 16:00:37 (1746129637) [ 467.349573] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 16:00:43 (1746129643) [ 473.970870] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 16:00:49 (1746129649) [ 486.799172] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 16:01:02 (1746129662) [ 490.197834] Lustre: *** cfs_fail_loc=513, val=17*** [ 490.199993] LustreError: 17254:0:(service.c:2266:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1830949203463040 [ 491.158086] Lustre: Failing over lustre-OST0000 [ 491.184554] LustreError: 32037:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 491.255659] Lustre: server umount lustre-OST0000 complete [ 491.287630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.293534] Lustre: Skipped 1 previous similar message [ 496.293885] LustreError: 9586:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.301081] LustreError: 9586:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 498.721855] LustreError: 9586:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.302111] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.310741] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 503.732927] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.455427] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 504.464004] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 504.514242] Lustre: 32575:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 504.584043] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 504.606337] mount.lustre (32575) used greatest stack depth: 9952 bytes left [ 506.141553] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 506.949013] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 507.373851] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 507.374348] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 510.168372] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 510.996047] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 516.164364] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 16:01:32 (1746129692) [ 519.299773] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 16:01:35 (1746129695) [ 522.729134] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 16:01:38 (1746129698) [ 526.656329] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 16:01:42 (1746129702) [ 530.982410] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 16:01:46 (1746129706) [ 534.083900] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 16:01:49 (1746129709) [ 537.436016] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 16:01:53 (1746129713) [ 541.270360] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 16:01:56 (1746129716) [ 541.421404] hrtimer: interrupt took 9836448 ns [ 544.614090] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 16:02:00 (1746129720) [ 548.094498] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 16:02:03 (1746129723) [ 552.070667] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 16:02:07 (1746129727) [ 553.110008] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.122898] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 553.131984] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 553.136857] Lustre: Skipped 1 previous similar message [ 553.999529] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 554.005446] Lustre: Skipped 1 previous similar message [ 554.412148] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.417706] Lustre: Skipped 2 previous similar messages [ 554.422056] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 554.425120] Lustre: Skipped 2 previous similar messages [ 563.739986] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 572.287708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.293771] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 572.296835] Lustre: Skipped 1 previous similar message [ 572.300361] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 572.313812] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 573.733680] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 573.809999] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 575.242953] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 575.349752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 584.631114] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 593.985820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.993019] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 594.015426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 594.033548] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 595.798066] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 595.899781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 597.766410] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 597.901133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 609.239775] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 619.218252] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.222840] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 619.225415] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 619.248633] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 621.086186] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 621.216243] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 623.449114] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 623.578418] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 634.656671] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 643.199657] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.206456] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 643.213906] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 643.220878] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 644.585442] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 644.661455] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 646.071802] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 646.145609] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 648.540802] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 16:03:44 (1746129824) [ 651.924626] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 16:03:47 (1746129827) [ 655.063581] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 16:03:50 (1746129830) [ 672.691013] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 16:04:08 (1746129848) [ 690.524940] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 16:04:26 (1746129866) [ 694.599203] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 16:04:30 (1746129870) [ 698.008333] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 16:04:33 (1746129873) [ 701.638555] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 16:04:37 (1746129877) [ 707.766916] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 16:04:43 (1746129883) [ 708.604101] Lustre: *** cfs_fail_loc=217, val=0*** [ 710.084925] Lustre: *** cfs_fail_loc=217, val=0*** [ 710.089633] Lustre: Skipped 1 previous similar message [ 714.991797] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 16:04:50 (1746129890) [ 723.200747] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 724.384227] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 16:04:59 (1746129899) [ 729.022592] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 16:05:04 (1746129904) [ 734.156922] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 16:05:09 (1746129909) [ 764.019800] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 16:05:39 (1746129939) [ 768.087033] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 16:05:43 (1746129943) [ 771.646213] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 16:05:47 (1746129947) [ 774.960483] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 16:05:50 (1746129950) [ 818.108901] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 16:06:34 (1746129994) [ 847.295804] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 16:07:02 (1746130022) [ 851.268156] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 16:07:06 (1746130026) [ 851.672285] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc1e:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum e64954d, server csum e64954c [ 853.658053] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 855.368415] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.18@tcp inode [0x200000407:0xc1e:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 604ee49 (type 1), server csum a1328a25 (type 1) [ 856.793739] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 858.050701] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.18@tcp inode [0x200000407:0xc1e:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 630a5dd9 (type 2), server csum a335ed3 (type 2) [ 859.397863] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 860.622545] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.18@tcp inode [0x200000407:0xc1e:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 155ba3b3 (type 4), server csum e64954c (type 4) [ 861.861991] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 863.071733] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.18@tcp inode [0x200000407:0xc1e:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 1e72ae24 (type 10), server csum 21f1af28 (type 10) [ 864.439715] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 866.890753] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 868.120839] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.18@tcp inode [0x200000407:0xc1e:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 3e66f823 (type 40), server csum 7ea0f858 (type 40) [ 868.138592] LustreError: Skipped 1 previous similar message [ 869.270382] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 871.689745] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 874.932862] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 16:07:30 (1746130050) [ 878.797666] Lustre: 27920:0:(tgt_handler.c:1935:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc1f:0x0]:[0-1048575]-5d1830ef-693b7e2 [ 878.814943] LustreError: dumping log to /tmp/lustre-log.1746130055.27920 [ 878.884127] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.18@tcp inode [0x200000407:0xc1f:0x0] object 0x280000401:3839 extent [0-1048575], client returned csum 5d1830ef (type 4), server csum 693b7e2 (type 4) [ 878.896291] LustreError: Skipped 1 previous similar message [ 889.070887] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 16:07:44 (1746130064) [ 889.256937] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 895.562413] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 16:07:51 (1746130071) [ 896.488512] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 896.667002] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 899.937191] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 899.947953] LustreError: Skipped 3 previous similar messages [ 907.471405] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 907.479055] LustreError: Skipped 3 previous similar messages [ 918.507059] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 918.515884] LustreError: Skipped 3 previous similar messages [ 935.524664] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 935.538259] LustreError: Skipped 4 previous similar messages [ 955.282893] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 971.753203] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 971.758196] LustreError: Skipped 14 previous similar messages [ 1013.786555] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1042.620078] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum de2bf73f, server csum de2bf73e [ 1042.636241] LustreError: Skipped 25 previous similar messages [ 1070.977379] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1126.717133] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1171.990021] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.18@tcp inode [0x200000407:0xc22:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum fec05011, server csum fec05010 [ 1171.995859] LustreError: Skipped 47 previous similar messages [ 1183.230694] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1238.936425] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1295.183791] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1298.310149] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 16:14:33 (1746130473) [ 1299.213552] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1301.050399] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1305.694817] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 16:14:41 (1746130481) [ 1306.184066] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1307.597881] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 1309.760598] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1313.978928] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1316.766864] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 16:14:52 (1746130492) [ 1317.401329] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1318.084985] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1319.494017] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1322.201844] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in IDLE state after 2 sec [ 1324.329592] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1325.235762] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in FULL state after 0 sec [ 1326.077872] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1328.002954] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1337.843595] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in IDLE state after 9 sec [ 1339.190992] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1339.856852] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in FULL state after 0 sec [ 1340.772118] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1342.441259] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1352.341621] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in IDLE state after 9 sec [ 1353.649161] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1354.346986] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in FULL state after 0 sec [ 1355.079950] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1356.977143] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1368.016727] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in IDLE state after 10 sec [ 1369.783319] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1370.504147] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in FULL state after 0 sec [ 1371.299217] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1373.200359] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1383.123020] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in IDLE state after 9 sec [ 1384.638021] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1385.227210] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in FULL state after 0 sec [ 1385.873612] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1387.321128] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1397.169249] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in IDLE state after 9 sec [ 1398.718577] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1399.427451] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in FULL state after 0 sec [ 1400.195393] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1401.598980] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1412.470266] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in IDLE state after 10 sec [ 1413.839902] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid 50 [ 1414.548982] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b34b2800.ost_server_uuid in FULL state after 0 sec [ 1417.759073] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1418.591595] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 16:16:34 (1746130594) [ 1421.110410] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 16:16:36 (1746130596) [ 1421.816290] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1422.374134] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1422.999316] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1423.552437] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1425.795446] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1426.476553] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 16:16:42 (1746130602) [ 1431.446685] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 16:16:47 (1746130607) [ 1434.767500] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 16:16:50 (1746130610) [ 1451.313297] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 16:17:07 (1746130627) [ 1455.415106] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 16:17:11 (1746130631) [ 1456.107058] Lustre: *** cfs_fail_loc=228, val=0*** [ 1459.167305] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 16:17:14 (1746130634) [ 1459.813713] Lustre: *** cfs_fail_loc=228, val=0*** [ 1462.770101] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 16:17:18 (1746130638) [ 1477.906100] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 16:17:33 (1746130653) [ 1480.947570] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 16:17:36 (1746130656) [ 1581.256796] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 16:19:16 (1746130756) [ 1595.907068] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 16:19:31 (1746130771) [ 1633.286759] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 16:20:09 (1746130809) [ 1837.288117] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 16:23:32 (1746131012) [ 1876.071130] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 16:24:11 (1746131051) [ 1881.140553] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 16:24:16 (1746131056) [ 1926.443454] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 16:25:02 (1746131102) [ 1934.036971] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 16:25:09 (1746131109) [ 1938.572284] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 16:25:14 (1746131114) [ 1972.997314] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 16:25:48 (1746131148) [ 1979.927371] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 16:25:55 (1746131155) [ 1984.124896] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 16:25:59 (1746131159) [ 1991.040614] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 16:26:06 (1746131166) [ 1995.453252] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 16:26:10 (1746131170) [ 2018.894341] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 16:26:34 (1746131194) [ 2042.862442] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 16:26:58 (1746131218) [ 2043.694366] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 2044.512770] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 2045.218071] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 2045.962691] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2048.808968] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 16:27:04 (1746131224) [ 2050.293740] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2051.316298] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2052.263925] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2053.294479] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2054.348816] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2058.331258] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 16:27:13 (1746131233) [ 2062.445186] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 16:27:17 (1746131237) [ 2083.249646] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 16:27:38 (1746131258) [ 2087.725620] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 16:27:43 (1746131263) [ 2091.950984] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 16:27:47 (1746131267) [ 2095.633800] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 16:27:51 (1746131271) [ 2099.419997] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 16:27:55 (1746131275) [ 2103.181576] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 16:27:58 (1746131278) [ 2107.202762] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 16:28:02 (1746131282) [ 2111.449751] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 16:28:07 (1746131287) [ 2115.747061] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 16:28:11 (1746131291) [ 2120.256039] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 16:28:15 (1746131295) [ 2280.454933] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 16:30:55 (1746131455) [ 2381.887068] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 16:32:37 (1746131557) [ 2385.955172] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 16:32:41 (1746131561) [ 2565.739796] Lustre: lustre-MDT0000: Client ec289b46-3a49-4bb7-a37a-0e153a5bc8d6 (at 192.168.203.18@tcp) reconnecting [ 2945.372315] Lustre: lustre-MDT0000: Client ec289b46-3a49-4bb7-a37a-0e153a5bc8d6 (at 192.168.203.18@tcp) reconnecting [ 3022.716204] Lustre: lustre-MDT0000: Client ec289b46-3a49-4bb7-a37a-0e153a5bc8d6 (at 192.168.203.18@tcp) reconnecting [ 3116.892392] Lustre: 9407:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 3116.895890] Lustre: 9407:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 3116.899545] Lustre: 9407:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 3116.903275] Lustre: 9407:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 3116.907702] Lustre: 9407:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 3116.911245] Lustre: 9407:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 3116.914701] CPU: 1 PID: 9407 Comm: mdt00_006 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 3116.919059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 3116.922445] Call Trace: [ 3116.923452] [] dump_stack+0x19/0x1b [ 3116.925709] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 3116.928322] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 3116.930703] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 3116.933094] [] lod_trans_start+0x7f/0x2e0 [lod] [ 3116.935781] [] mdd_trans_start+0x14/0x20 [mdd] [ 3116.937579] [] mdd_unlink+0x617/0xea0 [mdd] [ 3116.939157] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 3116.942028] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 3116.944210] [] mdt_reint_rec+0x87/0x240 [mdt] [ 3116.946352] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 3116.948841] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 3116.951225] [] mdt_reint+0x67/0x150 [mdt] [ 3116.953160] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 3116.955481] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 3116.958607] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 3116.960786] [] ? do_raw_spin_unlock+0x49/0x90 [ 3116.963672] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 3116.965766] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 3116.968199] [] kthread+0xe4/0xf0 [ 3116.969573] [] ? kthread_create_on_node+0x140/0x140 [ 3116.971523] [] ret_from_fork_nospec_begin+0x7/0x21 [ 3116.973773] [] ? kthread_create_on_node+0x140/0x140 [ 3120.152159] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 16:44:55 (1746132295) [ 3121.953589] Lustre: lustre-MDD0000: changelog on [ 3123.579707] Lustre: lustre-MDD0001: changelog on [ 3127.316748] Lustre: lustre-MDD0001: changelog off [ 3128.869583] Lustre: lustre-MDD0000: changelog off [ 3130.546757] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 16:45:05 (1746132305) [ 3130.833200] Lustre: lustre-OST0000: Client ec289b46-3a49-4bb7-a37a-0e153a5bc8d6 (at 192.168.203.18@tcp) reconnecting [ 3132.496580] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012e3f3800.ost_server_uuid 50 [ 3133.196809] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012e3f3800.ost_server_uuid in FULL state after 0 sec [ 3136.685745] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 16:45:12 (1746132312) [ 3139.764554] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 16:45:15 (1746132315) [ 3140.703376] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 3141.791279] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 16:45:17 (1746132317) [ 3145.365341] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 16:45:20 (1746132320) [ 3149.321907] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 16:45:24 (1746132324) [ 3152.870823] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 16:45:28 (1746132328) [ 3156.016223] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 16:45:31 (1746132331) [ 3169.785076] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 16:45:45 (1746132345) [ 3173.192062] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 16:45:48 (1746132348) [ 3177.550016] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 16:45:53 (1746132353) [ 3182.754741] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 16:45:58 (1746132358) [ 3186.712368] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 16:46:02 (1746132362) [ 3195.672214] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 16:46:11 (1746132371) [ 3199.476762] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 16:46:14 (1746132374) [ 3205.485722] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 16:46:20 (1746132380) [ 3209.901106] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 16:46:25 (1746132385) [ 3372.299403] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 16:49:07 (1746132547) [ 3374.066955] Lustre: *** cfs_fail_loc=147, val=0*** [ 3380.280971] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 16:49:15 (1746132555) [ 3384.615824] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 16:49:20 (1746132560) [ 3389.144058] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 16:49:24 (1746132564) [ 3390.305572] Lustre: *** cfs_fail_loc=217, val=0*** [ 3390.307826] Lustre: Skipped 21 previous similar messages [ 3394.789616] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3395.942062] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3397.008576] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 16:49:32 (1746132572) [ 3401.270421] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 16:49:36 (1746132576) [ 3405.337498] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 16:49:40 (1746132580) [ 3406.379677] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3407.418175] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3409.429926] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3412.446105] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3420.610100] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 16:49:56 (1746132596) [ 3421.818682] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3421.820553] Lustre: Skipped 1 previous similar message [ 3431.711158] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 16:50:07 (1746132607) [ 3432.873408] Lustre: *** cfs_fail_loc=220, val=0*** [ 3432.876908] Lustre: Skipped 2 previous similar messages [ 3437.231583] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 16:50:12 (1746132612) [ 3450.418460] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 16:50:26 (1746132626) [ 3454.017421] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 16:50:29 (1746132629) [ 3457.085340] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 16:50:32 (1746132632) [ 3459.910255] LustreError: 9589:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 3460.417715] LustreError: 9589:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout interrupted [ 3463.641141] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 16:50:39 (1746132639) [ 3466.457328] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 16:50:42 (1746132642) [ 3469.522551] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 16:50:45 (1746132645) [ 3473.010238] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 16:50:48 (1746132648) [ 3475.303062] Lustre: DEBUG MARKER: sanity test_119e: @@@@@@ FAIL: trivial unaligned dio failed [ 3479.152835] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 16:50:54 (1746132654) [ 3489.536938] Lustre: DEBUG MARKER: sanity test_119f: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3494.860228] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 16:51:10 (1746132670) [ 3508.737133] Lustre: DEBUG MARKER: sanity test_119g: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3513.752756] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 16:51:29 (1746132689) [ 3515.200343] Lustre: DEBUG MARKER: sanity test_119h: @@@@@@ FAIL: multiop memory unaligned write failed, 1044480 [ 3519.534959] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 3520.419994] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 16:51:36 (1746132696) [ 3521.241503] Lustre: DEBUG MARKER: SKIP: sanity test_119j needs kernel > 4.5.0 for ki_flags support [ 3522.448797] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 16:51:37 (1746132697) [ 3527.040410] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 16:51:42 (1746132702) [ 3528.278306] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 3529.604616] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 16:51:44 (1746132704) [ 3530.523178] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 3531.597928] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 16:51:47 (1746132707) [ 3532.618180] Lustre: DEBUG MARKER: sanity test_119p: @@@@@@ FAIL: DIO unaligned writev test failed [ 3535.718086] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 16:51:51 (1746132711) [ 3536.582821] Lustre: DEBUG MARKER: sanity test_119q: @@@@@@ FAIL: DIO unaligned writev test failed: 69631 [ 3539.337233] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 16:51:55 (1746132715) [ 3544.791062] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 16:52:00 (1746132720) [ 3549.923073] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 16:52:05 (1746132725) [ 3556.061718] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 16:52:11 (1746132731) [ 3561.468441] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 16:52:16 (1746132736) [ 3573.786981] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 16:52:29 (1746132749) [ 3586.088877] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 16:52:41 (1746132761) [ 3839.158296] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 16:56:55 (1746133015) [ 3841.818334] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 16:56:57 (1746133017) [ 3845.480080] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 3846.465049] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 3865.882957] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 12 sec [ 3868.677306] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 3934.722858] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 44 sec [ 3944.661586] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 8 sec [ 3945.410802] Lustre: DEBUG MARKER: ls -l done [ 3981.117703] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 35 seconds [ 3984.869410] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 16:59:20 (1746133160) [ 3985.557438] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 3986.415274] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 16:59:22 (1746133162) [ 3987.156599] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 3988.123611] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:59:23 (1746133163) [ 3993.129255] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 3994.350559] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 4014.031280] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 11 sec [ 4018.531884] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 4 sec [ 4086.317749] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 48 sec [ 4100.850387] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 13 sec [ 4101.694679] Lustre: DEBUG MARKER: ls -l done [ 4141.174264] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 38 seconds [ 4414.659549] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 4415.825625] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 4435.626745] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 12 sec [ 4438.937480] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 4508.429117] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 48 sec [ 4517.040934] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 7 sec [ 4517.736919] Lustre: DEBUG MARKER: ls -l done [ 4555.340980] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 37 seconds [ 4767.920573] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 17:12:23 (1746133943) [ 4780.899429] Lustre: DEBUG MARKER: ls done [ 4795.976839] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 17:12:51 (1746133971) [ 4802.100270] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 17:12:57 (1746133977) [ 4816.081917] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 17:13:11 (1746133991) [ 4893.172200] Lustre: 7246:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4965.438249] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 17:15:40 (1746134140) [ 5024.768334] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5024.771579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5134.940605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5135.487534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5149.949543] Lustre: 18934:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 5151.038679] Lustre: 7246:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 5206.556832] Lustre: 7245:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 5302.308858] Lustre: 7240:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 17472 > trans_max 3200 [ 5302.312371] Lustre: 7240:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 5302.317294] Lustre: 7240:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1143/1143/0, xattr_set: 1714/16108/0 [ 5302.321134] Lustre: 7240:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 5302.326073] Lustre: 7240:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 5302.328936] Lustre: 7240:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 5302.333488] CPU: 3 PID: 7240 Comm: mdt00_000 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 5302.340674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5302.346268] Call Trace: [ 5302.347864] [] dump_stack+0x19/0x1b [ 5302.349950] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 5302.352574] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 5302.356932] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 5302.361407] [] lod_trans_start+0x7f/0x2e0 [lod] [ 5302.364822] [] mdd_trans_start+0x14/0x20 [mdd] [ 5302.368455] [] mdd_unlink+0x617/0xea0 [mdd] [ 5302.370647] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 5302.374578] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 5302.378943] [] mdt_reint_rec+0x87/0x240 [mdt] [ 5302.381573] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 5302.385841] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 5302.388573] [] mdt_reint+0x67/0x150 [mdt] [ 5302.390861] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 5302.393781] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 5302.397730] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 5302.399745] [] ? _raw_spin_unlock_irq+0x15/0x30 [ 5302.404067] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 5302.407450] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 5302.410901] [] kthread+0xe4/0xf0 [ 5302.413857] [] ? kthread_create_on_node+0x140/0x140 [ 5302.416699] [] ret_from_fork_nospec_begin+0x7/0x21 [ 5302.421014] [] ? kthread_create_on_node+0x140/0x140 [ 5349.419239] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 17:22:04 (1746134524) [ 5378.149879] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 17:22:33 (1746134553) [ 6117.212393] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 17:34:52 (1746135292) [ 6206.794165] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 17:36:22 (1746135382) [ 6210.823959] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 17:36:26 (1746135386) [ 6211.690252] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 6212.517403] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 17:36:28 (1746135388) [ 6285.994442] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 17:37:41 (1746135461) [ 6286.856765] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 6306.266728] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff880098e1d000 [ 6307.082672] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff880098e1d000 [ 6307.877646] Lustre: DEBUG MARKER: LRU=1004 [ 6308.718348] Lustre: DEBUG MARKER: LIMIT=46624 [ 6309.490420] Lustre: DEBUG MARKER: LVF=5572500 [ 6310.196995] Lustre: DEBUG MARKER: OLD_LVF=100 [ 6310.865915] Lustre: DEBUG MARKER: Sleep 50 sec [ 6361.903269] Lustre: DEBUG MARKER: Dropped 414 locks in 50s [ 6362.798760] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 6377.323141] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 17:39:12 (1746135552) [ 6432.485775] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 6510.114603] Lustre: DEBUG MARKER: ls -la time: 76 seconds [ 6510.683867] Lustre: DEBUG MARKER: lru_size = 400 [ 6597.384289] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 6626.926174] Lustre: DEBUG MARKER: ls -la time: 28 seconds [ 6627.822262] Lustre: DEBUG MARKER: lru_size = 4052 [ 6628.602419] Lustre: DEBUG MARKER: ls -la is 63% faster with lru resize enabled [ 6664.952419] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 17:44:00 (1746135840) [ 6691.730966] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 17:44:27 (1746135867) [ 6718.457805] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 17:44:54 (1746135894) [ 6722.301979] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 17:44:57 (1746135897) [ 6725.532490] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 17:45:01 (1746135901) [ 6729.649836] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 17:45:05 (1746135905) [ 6733.208812] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 6749.692612] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 17:45:25 (1746135925) [ 6752.532992] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 17:45:28 (1746135928) [ 6760.344369] Lustre: 7242:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513271, FID: [0x20000040c:0x82de:0x0]) is approaching max size limit [ 6761.366921] Lustre: 7242:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513271, FID: [0x20000040c:0x82de:0x0]) is approaching max size limit [ 6767.460094] Lustre: 9423:0:(osd_handler.c:596:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513271, FID: [0x20000040c:0x82de:0x0]) has reached max size limit [ 6783.019064] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 17:45:58 (1746135958) [ 6786.665931] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 17:46:02 (1746135962) [ 6790.130342] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 17:46:05 (1746135965) [ 6793.487442] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 17:46:09 (1746135969) [ 6794.191578] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 6795.023124] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 17:46:10 (1746135970) [ 6819.022759] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 17:46:34 (1746135994) [ 6821.270597] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 17:46:37 (1746135997) [ 6833.313719] Lustre: 9407:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 6372 > trans_max 3200 [ 6833.318689] Lustre: 9407:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 6833.323750] Lustre: 9407:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5748/0 [ 6833.328479] Lustre: 9407:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 6833.334708] Lustre: 9407:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 6833.339393] Lustre: 9407:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 6833.344917] CPU: 0 PID: 9407 Comm: mdt00_006 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6833.351071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 6833.358176] Call Trace: [ 6833.359853] [] dump_stack+0x19/0x1b [ 6833.364661] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 6833.370175] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 6833.374000] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 6833.377883] [] lod_trans_start+0x7f/0x2e0 [lod] [ 6833.383142] [] mdd_trans_start+0x14/0x20 [mdd] [ 6833.387238] [] mdd_unlink+0x617/0xea0 [mdd] [ 6833.391525] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 6833.397162] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 6833.401053] [] mdt_reint_rec+0x87/0x240 [mdt] [ 6833.405716] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 6833.409290] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 6833.412686] [] mdt_reint+0x67/0x150 [mdt] [ 6833.415954] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 6833.421021] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 6833.425033] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 6833.427742] [] ? do_raw_spin_unlock+0x49/0x90 [ 6833.431349] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 6833.435101] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 6833.438799] [] kthread+0xe4/0xf0 [ 6833.441163] [] ? kthread_create_on_node+0x140/0x140 [ 6833.444629] [] ret_from_fork_nospec_begin+0x7/0x21 [ 6833.449956] [] ? kthread_create_on_node+0x140/0x140 [ 6835.873274] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 17:46:51 (1746136011) [ 6845.322974] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 17:47:01 (1746136021) [ 6866.892353] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 17:47:22 (1746136042) [ 6869.945747] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 17:47:25 (1746136045) [ 6873.003111] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 17:47:28 (1746136048) [ 6875.646626] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 17:47:31 (1746136051) [ 6878.967818] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 17:47:34 (1746136054) [ 6882.488866] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 17:47:38 (1746136058) [ 6892.445284] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 17:47:48 (1746136068) [ 6897.887719] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 17:47:53 (1746136073) [ 6915.540611] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 17:48:11 (1746136091) [ 6933.765279] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 17:48:29 (1746136109) [ 6939.947909] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 17:48:35 (1746136115) [ 6950.488587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6950.495354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.505726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.210652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.214160] Lustre: Skipped 3 previous similar messages [ 6954.333215] LustreError: 24862:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6954.336104] LustreError: 24862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6954.459920] Lustre: server umount lustre-MDT0000 complete [ 6956.215039] LustreError: 9417:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.224782] LustreError: 9417:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6958.542294] LustreError: 11687:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746136134 with bad export cookie 15890880697819432974 [ 6958.552526] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6958.561076] LustreError: 11687:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6958.647408] LustreError: 25461:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6958.653996] LustreError: 25461:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6958.816878] Lustre: server umount lustre-MDT0001 complete [ 6972.753723] LustreError: 26054:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6972.757079] LustreError: 26054:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6972.829537] Lustre: server umount lustre-OST0000 complete [ 6976.707762] Lustre: 3782:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746136137/real 1746136137] req@ffff88005ee17480 x1830949265830784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746136153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 6976.721022] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.734660] Lustre: Skipped 3 previous similar messages [ 6977.411542] LustreError: 26658:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6977.414242] LustreError: 26658:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6977.610275] Lustre: server umount lustre-OST0001 complete [ 6980.708680] device-mapper: core: cleaned up [ 6984.769073] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6985.893824] Key type lgssc unregistered [ 6986.017923] LNet: 27511:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6987.031220] LNet: Removed LNI 192.168.203.118@tcp [ 6993.477216] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6993.770743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6993.820546] alg: No test for adler32 (adler32-zlib) [ 6994.703041] Lustre: Lustre: Build Version: 2.16.54_86_g77fe414 [ 6994.793749] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6994.795243] LNet: Accept secure, port 988 [ 6996.334996] Key type lgssc registered [ 6996.844276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7003.903761] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7005.959639] device-mapper: uevent: version 1.0.3 [ 7005.962131] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 7009.055123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7009.075175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.367922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.732365] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7016.969074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7017.033114] Lustre: 30787:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7017.198331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7017.256258] mount.lustre (30787) used greatest stack depth: 9792 bytes left [ 7018.615843] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7019.855562] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7023.756908] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7023.765388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7023.876686] Lustre: 31911:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7023.973633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7024.000313] mount.lustre (31911) used greatest stack depth: 9584 bytes left [ 7026.329338] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7026.976010] LustreError: 31915:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.996072] LustreError: 31915:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7027.014516] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11476 to 0x280000400:11521) [ 7031.983957] LustreError: 31915:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.013973] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7032.020378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7032.135220] Lustre: 643:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7032.234806] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.676210] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7035.264003] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23047 to 0x280000403:23169) [ 7035.289214] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11641 to 0x2c0000400:11681) [ 7035.293838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22463 to 0x2c0000403:22529) [ 7040.522767] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7041.927843] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7050.270622] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 17:50:26 (1746136226) [ 7073.499074] LNet: 3397:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 7073.527344] Lustre: DEBUG MARKER:  [ 7073.529435] Lustre: DEBUG MARKER:  [ 7097.284848] LNet: 5022:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 7097.289684] LNet: 5022:0:(debug.c:388:cfs_str2mask()) Skipped 3 previous similar messages [ 7097.317734] Lustre: DEBUG MARKER:  [ 7097.319003] Lustre: DEBUG MARKER:  [ 7100.359394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.362846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7100.367718] Lustre: Skipped 3 previous similar messages [ 7105.367814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7105.374097] Lustre: Skipped 6 previous similar messages [ 7105.807955] LustreError: 5300:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7105.856467] Lustre: server umount lustre-MDT0000 complete [ 7109.799619] LustreError: 31935:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746136286 with bad export cookie 11461421456412939756 [ 7109.800500] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.813796] LustreError: 31935:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7109.958159] LustreError: 5900:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7109.964778] LustreError: 5900:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7110.068786] Lustre: server umount lustre-MDT0001 complete [ 7123.792197] LustreError: 6493:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7123.795400] LustreError: 6493:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7123.811990] Lustre: server umount lustre-OST0000 complete [ 7126.374264] Lustre: 27932:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746136286/real 1746136286] req@ffff88008625f800 x1830956481721984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746136302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 7126.387972] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.309014] LustreError: 7098:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7127.313013] LustreError: 7098:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7127.401673] Lustre: server umount lustre-OST0001 complete [ 7129.965436] device-mapper: core: cleaned up [ 7133.901342] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7134.923309] Key type lgssc unregistered [ 7135.074328] LNet: 8016:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7136.087690] LNet: Removed LNI 192.168.203.118@tcp [ 7141.768380] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7142.034692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7142.044024] alg: No test for adler32 (adler32-zlib) [ 7142.911144] Lustre: Lustre: Build Version: 2.16.54_86_g77fe414 [ 7142.989962] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7142.993214] LNet: Accept secure, port 988 [ 7144.532697] Key type lgssc registered [ 7144.925401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7149.590782] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7151.298076] device-mapper: uevent: version 1.0.3 [ 7151.299989] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 7154.133842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7154.149488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.392236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7156.551253] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7160.949721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7161.015449] Lustre: 11292:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7161.182899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7161.222746] mount.lustre (11292) used greatest stack depth: 9504 bytes left [ 7162.364646] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7163.567672] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7167.265046] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7167.273905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7167.408893] Lustre: 12420:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7167.550635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.698003] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7171.546386] LustreError: 12425:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7171.553354] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11476 to 0x280000400:11553) [ 7171.563714] LustreError: 12425:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7174.434362] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7174.443159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7174.538999] Lustre: 13613:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7174.670190] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7177.251729] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 7178.061645] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11641 to 0x2c0000400:11713) [ 7180.051700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23047 to 0x280000403:23201) [ 7180.052041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22463 to 0x2c0000403:22561) [ 7186.351797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7188.080457] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7196.726392] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 17:52:52 (1746136372) [ 7644.516053] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 18:00:20 (1746136820) [ 7655.027150] Lustre: *** cfs_fail_loc=327, val=500*** [ 7678.422640] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 18:00:53 (1746136853) [ 7682.332056] Lustre: *** cfs_fail_loc=328, val=500*** [ 7682.334966] Lustre: Skipped 1025 previous similar messages [ 7683.337726] Lustre: *** cfs_fail_loc=328, val=500*** [ 7683.339286] Lustre: Skipped 115 previous similar messages [ 7684.261705] Lustre: 10221:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800a8dd3100 x1830956632686848/t0(0) o101->34a2b231-08d0-494e-8753-e14e4097f347@192.168.203.18@tcp:51/0 lens 648/0 e 0 to 0 dl 1746136871 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7685.292541] Lustre: 10221:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800a8dd0700 x1830956632688000/t0(0) o101->34a2b231-08d0-494e-8753-e14e4097f347@192.168.203.18@tcp:52/0 lens 648/0 e 0 to 0 dl 1746136872 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7686.339732] Lustre: *** cfs_fail_loc=328, val=500*** [ 7686.342262] Lustre: Skipped 138 previous similar messages [ 7686.345065] Lustre: 10221:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88012d7a4700 x1830956632688768/t0(0) o101->34a2b231-08d0-494e-8753-e14e4097f347@192.168.203.18@tcp:53/0 lens 648/0 e 0 to 0 dl 1746136873 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7689.324914] Lustre: 10220:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88011d6a6300 x1830956632752768/t0(0) o101->34a2b231-08d0-494e-8753-e14e4097f347@192.168.203.18@tcp:56/0 lens 648/0 e 0 to 0 dl 1746136876 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7689.337535] Lustre: 10220:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 1 previous similar message [ 7690.359196] Lustre: *** cfs_fail_loc=328, val=500*** [ 7690.361687] Lustre: Skipped 124 previous similar messages [ 7693.761585] Lustre: 14778:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88008ea74a80 x1830956632786432/t0(0) o101->34a2b231-08d0-494e-8753-e14e4097f347@192.168.203.18@tcp:61/0 lens 648/0 e 0 to 0 dl 1746136881 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7693.779492] Lustre: 14778:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 3 previous similar messages [ 7698.880491] Lustre: *** cfs_fail_loc=328, val=500*** [ 7698.882038] Lustre: Skipped 66 previous similar messages [ 7702.072664] LustreError: 22482:0:(ldlm_lockd.c:3258:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 7710.772637] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 7711.547100] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 7712.354959] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 18:01:28 (1746136888) [ 7736.870535] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 18:01:52 (1746136912) [ 7758.195035] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 18:02:13 (1746136933) [ 7779.307009] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 18:02:34 (1746136954) [ 7799.398047] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 18:02:55 (1746136975) [ 7808.097792] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 18:03:03 (1746136983) [ 7818.891075] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 18:03:14 (1746136994) [ 7833.419593] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 18:03:28 (1746137008) [ 7843.357941] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 18:03:38 (1746137018) [ 7864.192362] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 18:03:59 (1746137039) [ 7865.385368] Lustre: DEBUG MARKER: SKIP: sanity test_150h Test must be statx() syscall supported [ 7866.351846] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 18:04:01 (1746137041) [ 7879.058233] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 18:04:14 (1746137054) [ 7888.567356] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 18:04:24 (1746137064) [ 7897.553068] bash (32535): drop_caches: 1 [ 7904.053972] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 18:04:39 (1746137079) [ 7907.340356] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 18:04:43 (1746137083) [ 7910.823618] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 18:04:46 (1746137086) [ 7914.336247] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 18:04:50 (1746137090) [ 7917.741039] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 18:04:53 (1746137093) [ 7918.745708] LustreError: 25080:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7923.610405] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 18:04:59 (1746137099) [ 7924.670618] LustreError: 14778:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7924.675646] LustreError: 14778:0:(fld_handler.c:244:fld_server_lookup()) Skipped 3 previous similar messages [ 7930.214391] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 18:05:05 (1746137105) [ 7934.458857] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 18:05:09 (1746137109) [ 7940.144127] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 18:05:15 (1746137115) [ 7943.819033] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 18:05:19 (1746137119) [ 7985.621856] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 18:06:00 (1746137160) [ 7990.772072] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 18:06:06 (1746137166) [ 8534.098886] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 18:15:09 (1746137709) [ 8537.071654] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 18:15:12 (1746137712) [ 8558.934749] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 18:15:34 (1746137734) [ 8564.751778] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 18:15:40 (1746137740) [ 8571.019535] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 18:15:46 (1746137746) [ 8577.219346] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 18:15:52 (1746137752) [ 8584.230803] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 18:15:59 (1746137759) [ 8608.501493] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 18:16:24 (1746137784) [ 8630.141760] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 18:16:45 (1746137805) [ 8656.141570] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 18:17:12 (1746137832) [ 8688.898961] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 18:17:44 (1746137864) [ 8693.002663] Lustre: DEBUG MARKER: Turn on read and write cache [ 8694.727969] Lustre: DEBUG MARKER: Write data and read it back. [ 8695.456428] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8697.358721] Lustre: DEBUG MARKER: cache hits: before: 65582, after: 65585 [ 8697.960066] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8699.077953] Lustre: DEBUG MARKER: cache hits:: before: 65585, after: 65588 [ 8699.602667] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 8701.232171] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8702.750894] Lustre: DEBUG MARKER: cache hits:: before: 65588, after: 65591 [ 8703.272564] Lustre: DEBUG MARKER: Write data and read it back. [ 8703.830293] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8705.471118] Lustre: DEBUG MARKER: cache hits:: before: 65591, after: 65594 [ 8706.187245] Lustre: DEBUG MARKER: Turn off read and write cache [ 8707.680210] Lustre: DEBUG MARKER: Write data and read it back [ 8708.251778] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8709.784608] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8710.338289] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 8711.814768] Lustre: DEBUG MARKER: Write data and read it back [ 8712.416982] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8713.981563] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8714.553119] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8716.118200] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65597 [ 8719.932390] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 18:18:15 (1746137895) [ 8721.263082] Lustre: lustre-MDD0000: changelog on [ 8722.521779] Lustre: lustre-MDD0001: changelog on [ 8730.737698] Lustre: Failing over lustre-MDT0000 [ 8733.442012] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 8733.911529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8733.918326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.703558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8735.705339] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.715548] Lustre: Skipped 4 previous similar messages [ 8736.479305] LustreError: 17246:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8736.541442] Lustre: server umount lustre-MDT0000 complete [ 8738.422481] LustreError: 25080:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.712275] LustreError: 11316:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.720628] LustreError: 11316:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 8740.952042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8741.034806] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.183732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8741.230089] Lustre: lustre-MDD0000: changelog on [ 8741.251068] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8743.025432] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 8743.431081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8746.185198] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 8746.492975] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8746.518577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:24036 to 0x280000403:24065) [ 8746.519043] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23402 to 0x2c0000403:23425) [ 8746.860331] Lustre: lustre-MDD0000: changelog off [ 8747.658904] Lustre: lustre-MDD0001: changelog off [ 8753.796386] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 18:18:49 (1746137929) [ 8755.230309] Lustre: lustre-MDD0000: changelog on [ 8760.674674] Lustre: lustre-MDD0001: changelog off [ 8762.926674] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 18:18:58 (1746137938) [ 8764.111901] Lustre: lustre-MDD0000: changelog on [ 8764.113722] Lustre: Skipped 1 previous similar message [ 8768.690821] Lustre: lustre-MDD0001: changelog off [ 8768.692562] Lustre: Skipped 1 previous similar message [ 8770.856061] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 18:19:06 (1746137946) [ 8773.172496] Lustre: lustre-MDD0001: changelog on [ 8773.174651] Lustre: Skipped 2 previous similar messages [ 8776.585364] Lustre: lustre-MDD0001: changelog off [ 8776.588677] Lustre: Skipped 1 previous similar message [ 8779.004432] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 18:19:14 (1746137954) [ 8786.510177] Lustre: lustre-MDD0001: changelog off [ 8786.512050] Lustre: Skipped 1 previous similar message [ 8788.778327] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 18:19:24 (1746137964) [ 8789.884895] Lustre: lustre-MDD0000: changelog on [ 8789.886578] Lustre: Skipped 2 previous similar messages [ 8793.931091] Lustre: DEBUG MARKER: 1746137969: creating first dirs [ 8810.515916] Lustre: *** cfs_fail_loc=1313, val=3*** [ 8810.518267] Lustre: 12049:0:(mdd_dir.c:911:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [ 8810.522672] Lustre: 12049:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8810.529823] Lustre: 25735:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 18s with 4 unprocessed records [ 8820.609269] Lustre: lustre-MDD0001: changelog off [ 8820.611414] Lustre: Skipped 1 previous similar message [ 8823.136217] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 18:19:58 (1746137998) [ 8824.573751] Lustre: lustre-MDD0000: changelog on [ 8824.575823] Lustre: Skipped 1 previous similar message [ 8835.170333] Lustre: 14778:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8835.175204] Lustre: 14778:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8835.183673] Lustre: 29029:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 8s with 4 unprocessed records [ 8835.189068] Lustre: 29029:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8846.355666] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 18:20:22 (1746138022) [ 8868.075298] Lustre: *** cfs_fail_loc=1316, val=0*** [ 8868.077092] Lustre: 3733:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 8868.080697] Lustre: 3733:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8868.084840] Lustre: 32322:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 17s with 3 unprocessed records [ 8868.090034] Lustre: 32322:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8868.741121] Lustre: Failing over lustre-MDT0000 [ 8868.799046] Lustre: Failing over lustre-MDT0001 [ 8871.304682] LustreError: 32633:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8871.308447] LustreError: 32633:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8871.332407] Lustre: server umount lustre-MDT0001 complete [ 8871.403955] Lustre: server umount lustre-MDT0000 complete [ 8874.059726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.120801] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.237982] LustreError: 12435:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8874.239302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8874.239304] Lustre: Skipped 2 previous similar messages [ 8874.252084] LustreError: 12435:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 8874.294736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.310985] Lustre: 1427:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 28 to index 29, being cleared now [ 8874.320397] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8875.326790] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 8877.969060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8878.023021] Lustre: 2312:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8878.098978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8878.103845] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 8878.107819] Lustre: Skipped 3 previous similar messages [ 8878.158524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8878.176517] Lustre: 2312:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 18 to index 19, being cleared now [ 8878.187385] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8878.683080] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8879.226165] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [ 8883.165087] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [ 8883.168451] Lustre: Skipped 1 previous similar message [ 8883.385390] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8883.408860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:24036 to 0x280000403:24097) [ 8883.408880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23427 to 0x2c0000403:23457) [ 8883.609087] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11719 to 0x2c0000400:11745) [ 8883.609409] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11558 to 0x280000400:11585) [ 8887.419583] Lustre: 8433:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746138047/real 1746138047] req@ffff880062c53b80 x1830956638540032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746138063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 8887.437663] Lustre: 8433:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8889.551065] Lustre: lustre-MDD0001: changelog off [ 8889.552903] Lustre: Skipped 3 previous similar messages [ 8891.843105] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 18:21:07 (1746138067) [ 8893.057363] Lustre: lustre-MDD0000: changelog on [ 8893.059511] Lustre: Skipped 5 previous similar messages [ 8896.930791] LustreError: 4870:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [ 8899.145610] LustreError: 5040:0:(mdd_device.c:1746:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [ 8899.150702] LustreError: 4870:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2783 [ 8899.941897] LustreError: 5289:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [ 8910.845864] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 18:21:26 (1746138086) [ 8922.172789] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 18:21:37 (1746138097) [ 8925.741523] LustreError: 7342:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [ 8928.749577] LustreError: 7342:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d awake [ 8938.207166] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 18:21:53 (1746138113) [ 8949.522669] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 18:22:05 (1746138125) [ 8957.323719] LustreError: 2321:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [ 8959.321878] LustreError: 7342:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [ 8959.330238] LustreError: 7342:0:(mdd_device.c:394:llog_changelog_cancel_cb()) Skipped 1 previous similar message [ 8959.337240] LustreError: 2321:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2992 [ 8964.194025] Lustre: lustre-MDD0001: changelog off [ 8964.195863] Lustre: Skipped 11 previous similar messages [ 8966.813461] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 18:22:22 (1746138142) [ 9992.517452] LustreError: 2321:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [ 9994.514709] LustreError: 5786:0:(llog_osd.c:1136:llog_osd_next_block()) cfs_fail_race id 16c waking [ 9994.518179] LustreError: 2321:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=3007 [10000.876025] Lustre: lustre-MDD0001: changelog off [10000.877429] Lustre: Skipped 1 previous similar message [10003.469429] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 18:39:39 (1746139179) [10004.809765] Lustre: lustre-MDD0000: changelog on [10004.811453] Lustre: Skipped 13 previous similar messages [10006.646781] LustreError: 16092:0:(mdd_device.c:1700:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [10007.086565] Lustre: 16157:0:(mdd_device.c:1717:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [10007.515244] LustreError: 16221:0:(mdd_device.c:1709:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [10020.313047] Lustre: lustre-MDD0000: changelog off [10020.314732] Lustre: Skipped 1 previous similar message [10023.525431] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 18:39:59 (1746139199) [10024.805605] Lustre: lustre-MDD0000: changelog on [10024.807452] Lustre: Skipped 1 previous similar message [10028.254535] Lustre: Failing over lustre-MDT0000 [10028.396701] LustreError: 18676:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [10028.402916] LustreError: 18676:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [10028.504302] Lustre: server umount lustre-MDT0000 complete [10029.502646] LustreError: 5786:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.023741] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10030.030170] LustreError: 12435:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.051524] Lustre: Skipped 6 previous similar messages [10033.619647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.690985] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10033.852289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.880765] Lustre: 19337:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90152 to index 18446744073709551615, being cleared now [10033.898883] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10034.482994] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10034.487162] Lustre: Skipped 1 previous similar message [10035.488241] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10038.861722] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [10038.865023] Lustre: Skipped 3 previous similar messages [10038.881519] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10038.886344] Lustre: Skipped 1 previous similar message [10038.916628] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23459 to 0x2c0000403:23489) [10038.916833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:24099 to 0x280000403:24129) [10044.621602] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 18:40:20 (1746139220) [10051.446902] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 18:40:27 (1746139227) [10060.837509] Lustre: 1436:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [10060.841277] Lustre: 1436:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [10060.854555] Lustre: 21924:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864007s with 500000004 unprocessed records [10060.869486] Lustre: 21924:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [10060.878663] Lustre: lustre-MDD0000: changelog off [10060.881145] Lustre: Skipped 3 previous similar messages [10069.902430] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 18:40:45 (1746139245) [10071.334363] Lustre: lustre-MDD0000: changelog on [10071.336148] Lustre: Skipped 4 previous similar messages [10100.530585] Lustre: *** cfs_fail_loc=18c, val=1211180*** [10100.537177] Lustre: Skipped 1 previous similar message [10100.538865] Lustre: 2321:0:(mdd_dir.c:881:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 21MB with 1MB space limit [10100.548340] Lustre: 2321:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [10100.554823] Lustre: 2321:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [10100.564620] Lustre: 24581:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 29s with 7505 unprocessed records [10100.577353] Lustre: 24581:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [10101.104435] Lustre: *** cfs_fail_loc=18c, val=1211180*** [10122.176441] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 18:41:37 (1746139297) [10129.301660] Lustre: lustre-MDD0001: changelog off [10129.305305] Lustre: Skipped 3 previous similar messages [10131.938771] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 18:41:47 (1746139307) [10150.449063] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 18:42:06 (1746139326) [10167.925801] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 18:42:23 (1746139343) [10169.143298] Lustre: lustre-MDD0000: changelog on [10169.145392] Lustre: Skipped 3 previous similar messages [10177.552675] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 18:42:33 (1746139353) [10187.972214] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 18:42:43 (1746139363) [10192.207328] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 18:42:47 (1746139367) [10196.286629] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 18:42:51 (1746139371) [10249.593192] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 18:43:45 (1746139425) [10256.155847] Lustre: Failing over lustre-OST0000 [10256.218735] LustreError: 31802:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10256.222594] LustreError: 31802:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10256.258410] Lustre: server umount lustre-OST0000 complete [10259.239067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.240405] LustreError: 3277:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10259.240407] LustreError: 3277:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [10259.255312] Lustre: Skipped 1 previous similar message [10263.929732] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10263.934590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10263.993673] Lustre: 473:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [10264.070896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10264.078332] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10264.851048] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10265.146752] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10265.146869] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [10265.146871] Lustre: Skipped 3 previous similar messages [10266.017807] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10268.130553] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 18:44:03 (1746139443) [10294.027456] Lustre: Failing over lustre-OST0000 [10294.052534] LustreError: 1899:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10294.054523] LustreError: 1899:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10294.079889] Lustre: server umount lustre-OST0000 complete [10294.121023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.133681] LustreError: 3265:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10294.152528] LustreError: 3265:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [10298.088252] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10298.095035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10298.178994] Lustre: 2509:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [10298.307119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10298.321439] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10299.397655] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10299.684138] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10299.685959] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [10299.685961] Lustre: Skipped 1 previous similar message [10300.748701] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10302.451222] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 18:44:38 (1746139478) [10317.121780] Lustre: Failing over lustre-OST0000 [10318.332124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.337601] Lustre: Skipped 1 previous similar message [10318.342721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10318.349605] Lustre: Skipped 1 previous similar message [10319.154571] LustreError: 3892:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10319.177269] LustreError: 3892:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10319.212062] Lustre: server umount lustre-OST0000 complete [10319.957341] LustreError: 3275:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.965413] LustreError: 3275:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [10323.004295] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10323.011865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10323.064832] Lustre: 4491:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [10323.189415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10323.199135] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10324.275114] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10324.600028] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10324.600756] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [10324.600758] Lustre: Skipped 1 previous similar message [10325.370881] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10327.199561] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 18:45:02 (1746139502) [10352.078716] Lustre: Failing over lustre-OST0000 [10352.105536] LustreError: 6932:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10352.112911] LustreError: 6932:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10352.136932] Lustre: server umount lustre-OST0000 complete [10353.223649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.232453] Lustre: Skipped 1 previous similar message [10353.236041] LustreError: 12426:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.243143] LustreError: 12426:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [10355.464443] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10355.471480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10355.526607] Lustre: 7529:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [10355.617291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10355.626420] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10357.343843] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10357.448987] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10357.495266] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [10357.496346] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10357.504033] Lustre: Skipped 1 previous similar message [10358.911227] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 18:45:34 (1746139534) [10371.453216] Lustre: Failing over lustre-OST0000 [10372.520683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10372.527185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.531794] Lustre: Skipped 1 previous similar message [10372.535136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.539236] Lustre: Skipped 1 previous similar message [10373.474523] LustreError: 8970:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10373.480762] LustreError: 8970:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10373.494066] Lustre: server umount lustre-OST0000 complete [10375.041682] LustreError: 21576:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10375.051494] LustreError: 21576:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [10376.571638] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10376.579987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10376.640785] Lustre: 9568:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [10376.723035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10376.732242] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10377.883087] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10378.604845] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10378.676019] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10378.676926] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [10378.676928] Lustre: Skipped 1 previous similar message [10380.077158] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 18:45:55 (1746139555) [10386.023799] Lustre: Failing over lustre-OST0000 [10386.072634] Lustre: server umount lustre-OST0000 complete [10392.248109] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10392.254488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10392.306221] Lustre: 11485:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [10393.963569] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10394.201723] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10395.386798] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 18:46:11 (1746139571) [10436.754430] Lustre: Failing over lustre-OST0000 [10436.778543] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10436.780840] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10436.793986] Lustre: server umount lustre-OST0000 complete [10437.466243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10437.473625] Lustre: Skipped 2 previous similar messages [10437.475990] LustreError: 3275:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10437.486867] LustreError: 3275:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 8 previous similar messages [10445.257173] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10445.262534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10445.323379] Lustre: 13572:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [10445.409561] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10445.418997] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10445.422790] Lustre: Skipped 1 previous similar message [10447.216572] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10447.228358] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [10447.228999] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10447.229000] Lustre: Skipped 1 previous similar message [10447.238547] Lustre: Skipped 3 previous similar messages [10447.400757] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [10448.928225] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 18:47:04 (1746139624) [10449.741030] Lustre: DEBUG MARKER: creating a 10 Mb file [10474.621840] Lustre: DEBUG MARKER: starting reads [10475.614594] Lustre: DEBUG MARKER: truncating the file [10476.717320] Lustre: DEBUG MARKER: killing dd [10477.468037] Lustre: DEBUG MARKER: removing the temporary file [10481.328579] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 18:47:36 (1746139656) [10485.231879] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 18:47:40 (1746139660) [10498.663809] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 18:47:54 (1746139674) [10505.146286] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 18:48:00 (1746139680) [10510.287233] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 18:48:05 (1746139685) [10511.050597] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [10511.994007] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 18:48:07 (1746139687) [10513.128304] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_module obdecho/obdecho [10522.587328] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 18:48:18 (1746139698) [10523.728890] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_module obdecho/obdecho [10523.829816] Lustre: Echo OBD driver; http://www.lustre.org/ [10534.689069] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 18:48:30 (1746139710) [10583.612756] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 18:49:19 (1746139759) [10610.957883] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 18:49:46 (1746139786) [10956.269794] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 18:55:31 (1746140131) [10956.964704] Lustre: *** cfs_fail_loc=148, val=0*** [10961.134824] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 18:55:36 (1746140136) [10967.450811] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 18:55:42 (1746140142) [10971.502913] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 18:55:47 (1746140147) [10995.595168] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 18:56:11 (1746140171) [11000.563320] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 18:56:16 (1746140176) [11005.691794] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 18:56:21 (1746140181) [11009.672513] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 18:56:25 (1746140185) [11014.499449] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 18:56:30 (1746140190) [11022.784338] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 18:56:38 (1746140198) [11028.467399] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 18:56:43 (1746140203) [11032.288940] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 18:56:47 (1746140207) [11054.776347] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 18:57:10 (1746140230) [11059.328214] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 18:57:14 (1746140234) [11063.639884] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 18:57:19 (1746140239) [11067.449961] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 18:57:22 (1746140242) [11071.169666] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 18:57:26 (1746140246) [11075.245876] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 18:57:30 (1746140250) [11079.229070] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 18:57:34 (1746140254) [11083.810378] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 18:57:39 (1746140259) [11087.869040] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 18:57:43 (1746140263) [11092.925906] Lustre: lustre-MDD0000: changelog on [11092.927763] Lustre: Skipped 3 previous similar messages [11096.226488] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [11096.975119] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.17401 [11098.219538] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [11098.974589] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.12842 [11100.319318] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [11101.193374] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.29052 [11102.537747] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [11103.358899] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.7100 [11104.748673] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [11105.441956] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.3258 [11106.611966] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [11107.395931] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.3784 [11108.722671] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11109.418894] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.31995 [11111.281434] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [11112.139661] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.81 [11113.725774] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [11114.518541] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.26392 [11116.007338] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [11116.872916] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.31429 [11119.081028] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [11119.975265] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.23271 [11121.457583] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [11122.312994] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.23048 [11127.758116] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11128.651409] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg318-client.v [11130.089275] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11131.092312] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg318-client.E [11132.513526] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11133.374124] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg318-client.v [11137.853218] Lustre: lustre-MDD0001: changelog off [11137.854792] Lustre: Skipped 5 previous similar messages [11143.608708] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 18:58:39 (1746140319) [11150.414977] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 18:58:45 (1746140325) [11154.787978] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 18:58:49 (1746140329) [11164.726725] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 18:59:00 (1746140340) [11175.307599] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 18:59:10 (1746140350) [11181.147837] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 18:59:16 (1746140356) [11277.852738] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 19:00:53 (1746140453) [11285.255123] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 19:01:00 (1746140460) [11292.704717] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 19:01:08 (1746140468) [11298.512493] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 19:01:14 (1746140474) [11376.168680] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 19:02:31 (1746140551) [11380.222017] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 19:02:35 (1746140555) [11384.814203] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 19:02:39 (1746140559) [11389.075976] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 19:02:44 (1746140564) [11396.411576] Lustre: Failing over lustre-MDT0000 [11396.455489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11396.458833] LustreError: Skipped 1 previous similar message [11396.460646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.467619] Lustre: Skipped 1 previous similar message [11396.469988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.600399] LustreError: 26370:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11396.606179] LustreError: 26370:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11396.726395] Lustre: server umount lustre-MDT0000 complete [11397.047892] LustreError: 1442:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.067752] LustreError: 1442:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [11405.839019] LustreError: 23531:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.856723] LustreError: 23531:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [11410.531462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11410.611325] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11410.775970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11410.780662] Lustre: Skipped 1 previous similar message [11410.806843] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11410.837969] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11412.270183] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [11415.788985] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [11415.796410] Lustre: Skipped 1 previous similar message [11415.823934] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11415.845641] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:37014 to 0x2c0000403:37057) [11415.846908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:37749 to 0x280000403:37793) [11417.343805] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11418.021264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11422.349549] Lustre: Failing over lustre-MDT0000 [11422.459787] LustreError: 27655:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11422.464828] LustreError: 27655:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11422.509800] Lustre: server umount lustre-MDT0000 complete [11425.799507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11425.800262] LustreError: 1437:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.800264] LustreError: 1437:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [11425.830382] Lustre: Skipped 5 previous similar messages [11436.329573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.400385] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11436.554129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11436.594394] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11438.103035] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [11440.886343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11441.564227] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [11441.568538] Lustre: Skipped 3 previous similar messages [11441.579940] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11441.605722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:37014 to 0x2c0000403:37089) [11441.606693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:37749 to 0x280000403:37825) [11443.051987] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11443.708625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11448.920803] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 19:03:44 (1746140624) [11462.568523] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 19:03:58 (1746140638) [11467.657859] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 19:04:03 (1746140643) [11473.384255] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 19:04:09 (1746140649) [11486.755822] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 19:04:22 (1746140662) [11505.067617] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 19:04:40 (1746140680) [11508.455940] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 19:04:44 (1746140684) [11518.631112] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 19:04:54 (1746140694) [11522.237371] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 19:04:58 (1746140698) [11522.936005] Lustre: DEBUG MARKER: creating a 10 Mb file [11547.581936] Lustre: DEBUG MARKER: starting reads [11548.265719] Lustre: DEBUG MARKER: truncating the file [11549.121703] Lustre: DEBUG MARKER: killing dd [11549.802975] Lustre: DEBUG MARKER: removing the temporary file [11553.221571] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 19:05:28 (1746140728) [11556.756326] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 19:05:32 (1746140732) [11559.366635] Lustre: *** cfs_fail_loc=229, val=0*** [11561.830074] Lustre: *** cfs_fail_loc=229, val=1*** [11563.287356] Lustre: *** cfs_fail_loc=229, val=1*** [11563.289619] Lustre: Skipped 1 previous similar message [11575.105534] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 19:05:50 (1746140750) [11581.042681] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 19:05:56 (1746140756) [11585.360726] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 19:06:00 (1746140760) [11589.453015] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 19:06:05 (1746140765) [11593.468350] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 19:06:09 (1746140769) [11593.666301] Lustre: lustre-OST0000: Client 420e42c8-d6e7-4cf1-810a-263c926a79ea (at 192.168.203.18@tcp) reconnecting [11597.736945] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 19:06:13 (1746140773) [11600.245548] LustreError: 10662:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 3000ms [11603.253560] LustreError: 10662:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 awake [11608.577998] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 19:06:23 (1746140783) [11616.728318] Lustre: *** cfs_fail_loc=520, val=57344*** [11616.781731] LNet: *** cfs_fail_loc=e000, val=2147483648*** [11621.760259] Lustre: lustre-OST0000: Client 420e42c8-d6e7-4cf1-810a-263c926a79ea (at 192.168.203.18@tcp) reconnecting [11631.374945] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 19:06:46 (1746140806) [11633.770156] LustreError: 24230:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 22000ms [11653.868832] Lustre: lustre-OST0000: Client 420e42c8-d6e7-4cf1-810a-263c926a79ea (at 192.168.203.18@tcp) reconnecting [11655.530493] LustreError: 24230:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout interrupted [11655.541847] LustreError: 24230:0:(ldlm_lib.c:3490:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff8800a8e2b100 x1830956720889984/t0(0) o3->420e42c8-d6e7-4cf1-810a-263c926a79ea@192.168.203.18@tcp:0/0 lens 488/440 e 0 to 0 dl 1746140830 ref 1 fl Interpret:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11655.554940] Lustre: 24230:0:(service.c:2494:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@ffff8800a8e2b100 x1830956720889984/t0(0) o3->420e42c8-d6e7-4cf1-810a-263c926a79ea@192.168.203.18@tcp:0/0 lens 488/440 e 0 to 0 dl 1746140830 ref 1 fl Complete:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11659.350897] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [11660.439519] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [11661.307391] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 19:07:16 (1746140836) [11665.062924] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 19:07:20 (1746140840) [11668.569958] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 19:07:24 (1746140844) [11672.436287] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 19:07:28 (1746140848) [11676.060949] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 19:07:31 (1746140851) [11679.557338] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 19:07:35 (1746140855) [11683.378143] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 19:07:38 (1746140858) [11811.750258] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 19:09:47 (1746140987) [11928.103778] Lustre: Failing over lustre-MDT0000 [11929.571273] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [11930.539210] LustreError: 9277:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11930.542209] LustreError: 9277:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11930.634585] Lustre: server umount lustre-MDT0000 complete [11931.384523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11931.389420] LustreError: 21660:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.389422] LustreError: 21660:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 14 previous similar messages [11931.420558] Lustre: Skipped 3 previous similar messages [11935.466143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11935.605005] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11935.852829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11935.899615] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11937.876845] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [11939.580108] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11940.842156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [11940.845420] Lustre: Skipped 3 previous similar messages [11940.855037] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11940.880885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:44038 to 0x280000403:44065) [11940.881323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:43268 to 0x2c0000403:43297) [11968.226305] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 19:12:23 (1746141143) [12238.876824] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 19:16:54 (1746141414) [12242.834080] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 19:16:58 (1746141418) [12247.323367] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 19:17:02 (1746141422) [12282.003270] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 19:17:37 (1746141457) [12283.342420] Lustre: *** cfs_fail_loc=1801, val=0*** [12283.351823] LustreError: 12975:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b78:0x567e:0x0]/f0 failed: rc = -5 [12284.393579] LustreError: 12975:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration: rc = -1 [12284.401371] LustreError: 12975:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200003ab1:0x2fea:0x0]/migrate_dir failed: rc = -1 [12286.878603] Lustre: 12975:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0001: [0x240001b71:0x16:0x0]/foo is open, migrate only dentry [12291.493420] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [12292.290273] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 19:17:48 (1746141468) [12296.408408] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 19:17:52 (1746141472) [12301.174486] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 19:17:57 (1746141477) [12304.171956] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 19:17:59 (1746141479) [12307.502819] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 19:18:03 (1746141483) [12311.006570] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 19:18:06 (1746141486) [12314.056516] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 19:18:09 (1746141489) [12314.771417] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [12315.655779] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 19:18:11 (1746141491) [12368.087214] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 19:19:03 (1746141543) [12373.855103] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 19:19:09 (1746141549) [12377.433270] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 19:19:13 (1746141553) [12392.360069] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 19:19:28 (1746141568) [12415.628059] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 19:19:51 (1746141591) [12440.843617] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 19:20:16 (1746141616) [12444.966148] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 19:20:20 (1746141620) [12450.444746] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 19:20:26 (1746141626) [12454.876916] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 19:20:30 (1746141630) [12455.584710] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [12456.481778] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 19:20:32 (1746141632) [12457.112265] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [12457.913845] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 19:20:33 (1746141633) [12461.930555] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 19:20:37 (1746141637) [12468.539956] LustreError: 12975:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000007:0x1:0x0]/d230x.sanity failed: rc = -28 [12468.544484] LustreError: 12975:0:(mdt_reint.c:2523:mdt_reint_migrate()) Skipped 1 previous similar message [12492.730037] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 19:21:08 (1746141668) [12494.961574] Lustre: *** cfs_fail_loc=1802, val=0*** [12495.035329] LustreError: 12975:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b78:0x5bda:0x0]/d97 failed: rc = -77 [12501.303570] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 19:21:17 (1746141677) [12503.378295] Lustre: *** cfs_fail_loc=1802, val=0*** [12503.381208] Lustre: Skipped 1 previous similar message [12503.454531] LustreError: 1451:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b78:0x5c11:0x0]/d57 failed: rc = -77 [12527.890407] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 19:21:43 (1746141703) [12532.961326] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 19:21:48 (1746141708) [12559.624963] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 19:22:15 (1746141735) [12560.107972] Lustre: *** cfs_fail_loc=31c, val=0*** [12561.900688] Lustre: Failing over lustre-OST0000 [12562.009061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12562.016962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12562.024630] Lustre: Skipped 1 previous similar message [12562.028164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12562.048572] LustreError: 22707:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [12562.051744] LustreError: 22707:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12562.106908] Lustre: server umount lustre-OST0000 complete [12566.072358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12566.083172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12566.096344] LustreError: 21575:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12566.107668] LustreError: 21575:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [12566.324808] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12566.333880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12566.415738] Lustre: 23303:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [12566.548146] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12566.561177] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12567.935332] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12568.406411] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [12568.406546] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [12568.412934] Lustre: Skipped 3 previous similar messages [12569.023259] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [12573.261984] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 19:22:28 (1746141748) [12573.897281] Lustre: *** cfs_fail_loc=31c, val=0*** [12573.898761] LustreError: 21575:0:(ldlm_request.c:475:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff880086b7a800/0xf188cfd7761d2e8a lrc: 2/0,0 mode: --/PR res: [0x280000403:0xae57:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 21575 timeout: 0 lvb_type: 0 [12576.024281] Lustre: Failing over lustre-OST0000 [12576.055569] LustreError: 24349:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [12576.058334] LustreError: 24349:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12576.073631] Lustre: server umount lustre-OST0000 complete [12576.558003] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12580.246186] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12580.253509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12580.334861] Lustre: 24944:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [12580.460545] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12580.472674] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12582.457049] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [12582.462357] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [12582.468720] Lustre: Skipped 1 previous similar message [12583.001315] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [12587.740703] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 19:22:43 (1746141763) [12591.791025] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 19:22:47 (1746141767) [12595.056226] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 19:22:50 (1746141770) [12598.017174] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 19:22:53 (1746141773) [12603.170878] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 19:22:58 (1746141778) [12607.133253] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 19:23:02 (1746141782) [12611.195502] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 19:23:06 (1746141786) [12672.224413] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 19:24:08 (1746141848) [12673.884244] Lustre: *** cfs_fail_loc=2100, val=0*** [12679.572398] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 19:24:15 (1746141855) [12686.637278] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 19:24:22 (1746141862) [12688.176333] Lustre: *** cfs_fail_loc=713, val=1*** [12694.825078] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 19:24:30 (1746141870) [12743.013274] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 19:25:18 (1746141918) [12765.229722] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 19:25:40 (1746141940) [12765.930049] Lustre: *** cfs_fail_loc=105, val=0*** [12765.932038] Lustre: Skipped 1 previous similar message [12769.451195] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 19:25:45 (1746141945) [12956.939481] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 19:28:52 (1746142132) [13019.758678] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 19:29:55 (1746142195) [13023.263618] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 19:29:59 (1746142199) [13026.382379] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 19:30:02 (1746142202) [13030.409649] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 19:30:06 (1746142206) [13035.114184] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 19:30:10 (1746142210) [13038.478238] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 19:30:14 (1746142214) [13043.328985] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 19:30:19 (1746142219) [13047.413747] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 19:30:23 (1746142223) [13050.502324] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 19:30:26 (1746142226) [13051.199583] Lustre: enable_remote_subdir_mount is deprecated, it's always enabled. [13058.196642] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 19:30:33 (1746142233) [13062.601087] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 19:30:38 (1746142238) [13069.475417] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 19:30:45 (1746142245) [13132.737284] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 19:31:48 (1746142308) [13160.568993] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 19:32:16 (1746142336) [13170.869957] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 19:32:26 (1746142346) [13173.369697] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 19:32:29 (1746142349) [13175.981437] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 19:32:31 (1746142351) [13179.322114] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 19:32:35 (1746142355) [13187.934403] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 19:32:43 (1746142363) [13193.700308] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 19:32:49 (1746142369) [13230.839056] LustreError: 5786:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200004283:0x3d:0x0]: have 0 want 1 [13270.910355] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 19:34:06 (1746142446) [13272.744857] Lustre: lustre-MDD0000: changelog on [13272.746627] Lustre: Skipped 1 previous similar message [13278.952610] Lustre: lustre-MDD0001: changelog off [13278.954347] Lustre: Skipped 1 previous similar message [13281.876109] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [13283.201630] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [13284.363024] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [13285.328789] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [13286.326935] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 19:34:21 (1746142461) [13287.188378] Lustre: *** cfs_fail_loc=161, val=0*** [13288.031951] Lustre: Failing over lustre-MDT0001 [13288.136713] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13288.142959] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13288.153790] Lustre: Skipped 1 previous similar message [13288.160106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13290.769673] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [13291.640752] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13291.651762] Lustre: Skipped 1 previous similar message [13293.760707] LustreError: 9078:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [13293.766338] LustreError: 9078:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13293.885842] Lustre: server umount lustre-MDT0001 complete [13295.748149] LustreError: 1438:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13295.756307] LustreError: 1438:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [13298.066680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13298.128628] Lustre: 9678:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13298.334242] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13298.358917] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [13299.855076] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13300.115862] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [13300.120205] Lustre: Skipped 1 previous similar message [13302.070057] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13303.347440] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [13303.360675] Lustre: Skipped 1 previous similar message [13303.378696] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [13303.406923] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:34781 to 0x2c0000400:34817) [13303.420542] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:34702 to 0x280000400:34721) [13310.353799] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 19:34:46 (1746142486) [13314.102893] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 19:34:49 (1746142489) [13318.129178] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 19:34:53 (1746142493) [13328.489569] Lustre: *** cfs_fail_loc=2301, val=0*** [13329.809936] Lustre: Failing over lustre-OST0000 [13329.838216] LustreError: 12056:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13329.841622] LustreError: 12056:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13329.921253] Lustre: server umount lustre-OST0000 complete [13332.886503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13332.890352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13333.929010] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13333.941210] LDISKFS-fs (dm-2): 1 truncate cleaned up [13333.942898] LDISKFS-fs (dm-2): recovery complete [13333.948078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13334.004156] Lustre: 12729:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13334.123741] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13334.134342] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13335.718894] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13336.483829] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13336.846307] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [13336.846678] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13336.862635] Lustre: Skipped 2 previous similar messages [13343.420576] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 19:35:19 (1746142519) [13346.797991] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 19:35:22 (1746142522) [13355.344971] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 19:35:30 (1746142530) [13359.312841] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 19:35:34 (1746142534) [13364.112066] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 19:35:39 (1746142539) [13368.059217] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 19:35:43 (1746142543) [13373.879923] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 19:35:49 (1746142549) [13376.882040] Lustre: Increasing provided stripe size to a minimum value 64 [13382.162481] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 19:35:57 (1746142557) [13385.951438] Lustre: *** cfs_fail_loc=168, val=40*** [13386.880348] Lustre: *** cfs_fail_loc=168, val=20*** [13386.891899] Lustre: Skipped 2 previous similar messages [13387.924295] Lustre: *** cfs_fail_loc=168, val=0*** [13387.925742] Lustre: Skipped 2 previous similar messages [13390.869538] Lustre: *** cfs_fail_loc=168, val=55*** [13390.874540] Lustre: Skipped 8 previous similar messages [13395.838859] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 19:36:11 (1746142571) [13401.820938] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 19:36:17 (1746142577) [13405.304728] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 19:36:20 (1746142580) [13410.514682] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 19:36:26 (1746142586) [13415.007174] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 19:36:30 (1746142590) [13419.501387] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 19:36:34 (1746142594) [13424.524228] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 19:36:39 (1746142599) [13509.091435] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 19:38:04 (1746142684) [13512.548326] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 19:38:08 (1746142688) [13516.073292] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 19:38:11 (1746142691) [13520.171428] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 19:38:15 (1746142695) [13524.363920] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 19:38:19 (1746142699) [13531.035845] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 19:38:26 (1746142706) [13537.166102] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 19:38:32 (1746142712) [13543.456964] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 19:38:39 (1746142719) [13549.911157] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 19:38:45 (1746142725) [13555.763931] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 19:38:51 (1746142731) [13560.562233] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 19:38:56 (1746142736) [13562.475344] LustreError: 1450:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b sleeping for 2000ms [13564.483595] LustreError: 1450:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b awake [13568.278257] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 19:39:03 (1746142743) [13572.973351] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 19:39:08 (1746142748) [13579.034834] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 19:39:14 (1746142754) [13580.358314] Lustre: Failing over lustre-OST0000 [13580.360990] LustreError: 25461:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13580.431363] LustreError: 25457:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13580.437127] LustreError: 25457:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13580.501672] Lustre: server umount lustre-OST0000 complete [13581.274989] LustreError: 3273:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13581.281009] LustreError: 3273:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [13582.246750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13582.250048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13582.265133] Lustre: Skipped 1 previous similar message [13584.849063] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13584.857189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13584.933300] Lustre: 26696:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13585.083044] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13585.095825] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13586.261819] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13586.817344] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13586.819778] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [13586.819779] Lustre: Skipped 1 previous similar message [13588.085900] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13589.789299] Lustre: Failing over lustre-OST0000 [13589.792795] LustreError: 28023:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13589.799655] LustreError: 28023:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13589.832283] LustreError: 28016:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13589.836465] LustreError: 28016:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13589.881038] Lustre: server umount lustre-OST0000 complete [13593.988087] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13593.995478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13594.094899] Lustre: 29246:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13594.148389] LustreError: 29279:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13594.266242] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13594.278842] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13595.627763] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13595.628514] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [13595.628516] Lustre: Skipped 1 previous similar message [13596.926653] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13598.898066] Lustre: Failing over lustre-OST0000 [13598.920203] LustreError: 30569:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13598.927025] LustreError: 30569:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13598.958802] Lustre: server umount lustre-OST0000 complete [13599.273569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13599.279335] Lustre: Skipped 4 previous similar messages [13603.090443] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13603.098917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13603.174838] Lustre: 31796:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13603.217150] LustreError: 31827:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13603.295140] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13603.309877] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13604.735678] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13604.741151] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [13604.759607] Lustre: Skipped 1 previous similar message [13606.042764] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13607.897970] Lustre: Failing over lustre-OST0000 [13607.963832] LustreError: 678:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13607.971443] LustreError: 678:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13608.018367] Lustre: server umount lustre-OST0000 complete [13612.526477] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13612.544038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13612.665586] LustreError: 1951:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13612.772880] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13612.800327] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13614.384043] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13614.390036] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [13614.390037] Lustre: Skipped 1 previous similar message [13615.959232] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13618.360937] Lustre: Failing over lustre-OST0000 [13618.483510] Lustre: server umount lustre-OST0000 complete [13619.400285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13619.409170] LustreError: Skipped 1 previous similar message [13619.411133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13619.419932] Lustre: Skipped 2 previous similar messages [13623.238392] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13623.255795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13623.315121] Lustre: 4434:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13623.319242] Lustre: 4434:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [13623.353336] LustreError: 4461:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13623.356974] LustreError: 4461:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [13624.812972] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13624.818689] Lustre: Skipped 3 previous similar messages [13626.678159] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13628.896321] Lustre: Failing over lustre-OST0000 [13628.957723] LustreError: 5758:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13628.959682] LustreError: 5758:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13629.060043] Lustre: server umount lustre-OST0000 complete [13630.422119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13630.427986] LustreError: Skipped 2 previous similar messages [13633.578915] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13633.590399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13633.846483] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13633.853317] Lustre: Skipped 1 previous similar message [13633.860135] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13633.864958] Lustre: Skipped 1 previous similar message [13635.632065] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13635.633063] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [13635.633065] Lustre: Skipped 3 previous similar messages [13635.649253] Lustre: Skipped 1 previous similar message [13636.232509] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13637.901906] Lustre: Failing over lustre-OST0000 [13637.975297] Lustre: server umount lustre-OST0000 complete [13642.698530] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13642.716755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13646.341224] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13648.595232] Lustre: Failing over lustre-OST0000 [13648.813011] Lustre: server umount lustre-OST0000 complete [13649.980731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13654.175131] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13654.182650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13656.932264] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13659.007045] Lustre: Failing over lustre-OST0000 [13659.124551] Lustre: server umount lustre-OST0000 complete [13659.352023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13659.361450] Lustre: Skipped 7 previous similar messages [13663.851017] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13663.865869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13663.999665] Lustre: 14630:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13664.003453] Lustre: 14630:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 3 previous similar messages [13666.692048] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13668.675193] Lustre: Failing over lustre-OST0000 [13668.675576] LustreError: 15921:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13668.675578] LustreError: 15921:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [13668.761532] LustreError: 15917:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13668.767717] LustreError: 15917:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [13668.838502] Lustre: server umount lustre-OST0000 complete [13673.775828] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13673.784412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13674.044700] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13674.048759] Lustre: Skipped 3 previous similar messages [13674.093321] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13674.095807] Lustre: Skipped 3 previous similar messages [13675.709574] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13675.710080] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.118@tcp (at 0@lo) [13675.710081] Lustre: Skipped 7 previous similar messages [13675.721201] Lustre: Skipped 3 previous similar messages [13676.230867] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13678.004147] Lustre: Failing over lustre-OST0000 [13678.068882] Lustre: server umount lustre-OST0000 complete [13681.694143] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13681.709764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13684.182263] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13686.055916] Lustre: Failing over lustre-OST0000 [13686.131910] Lustre: server umount lustre-OST0000 complete [13690.524124] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13690.531302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13691.446352] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13691.449766] Lustre: Skipped 6 previous similar messages [13693.199741] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13694.952506] Lustre: Failing over lustre-OST0000 [13695.067357] Lustre: server umount lustre-OST0000 complete [13698.905000] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13698.917095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13701.442565] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13703.376720] Lustre: Failing over lustre-OST0000 [13703.380683] LustreError: 26067:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13703.384233] LustreError: 26067:0:(obd_class.h:479:obd_check_dev()) Skipped 6 previous similar messages [13703.545175] Lustre: server umount lustre-OST0000 complete [13708.313774] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13708.326758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13710.702181] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13712.323700] Lustre: Failing over lustre-OST0000 [13712.422740] Lustre: server umount lustre-OST0000 complete [13716.415906] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13716.426683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13718.672400] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13720.426808] Lustre: Failing over lustre-OST0000 [13720.496221] Lustre: server umount lustre-OST0000 complete [13724.737343] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13724.743255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13727.203962] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13728.907346] Lustre: Failing over lustre-OST0000 [13728.983490] Lustre: server umount lustre-OST0000 complete [13729.975568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13729.984626] Lustre: Skipped 14 previous similar messages [13733.405067] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13733.421019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13733.511696] Lustre: 2414:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.118@tcp: no remote llog for lustre-sptlrpc, check MGS config [13733.517721] Lustre: 2414:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 7 previous similar messages [13735.708645] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13737.406901] Lustre: Failing over lustre-OST0000 [13737.453316] LustreError: 3734:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13737.456032] LustreError: 3734:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [13737.475347] Lustre: server umount lustre-OST0000 complete [13741.263447] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13741.271273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13741.443298] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13741.447013] Lustre: Skipped 7 previous similar messages [13741.451321] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13741.454034] Lustre: Skipped 7 previous similar messages [13742.534257] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13742.534813] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.118@tcp (at 0@lo) [13742.534814] Lustre: Skipped 15 previous similar messages [13742.548110] Lustre: Skipped 7 previous similar messages [13743.602317] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13745.542504] Lustre: Failing over lustre-OST0000 [13745.633679] Lustre: server umount lustre-OST0000 complete [13749.102001] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13749.108883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13751.304669] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13753.170200] Lustre: Failing over lustre-OST0000 [13753.250411] Lustre: server umount lustre-OST0000 complete [13756.819870] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13756.827790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13758.939328] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13763.096858] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 19:42:18 (1746142938) [13767.014994] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 19:42:22 (1746142942) [13768.764079] Lustre: Failing over lustre-MDT0000 [13768.842957] Lustre: Failing over lustre-MDT0001 [13769.165519] LustreError: 12137:0:(tgt_mount.c:1756:server_put_super()) cfs_race id 60c sleeping [13769.244220] Lustre: server umount lustre-MDT0000 complete [13773.600963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13773.747870] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13773.760376] LustreError: 12728:0:(tgt_mount.c:1486:server_start_targets()) cfs_fail_race id 60c waking [13773.764363] LustreError: 12137:0:(tgt_mount.c:1756:server_put_super()) cfs_fail_race id 60c awake: rc=413 [13773.797953] Lustre: server umount lustre-MDT0001 complete [13775.949531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13777.856179] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13783.472181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13783.707340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [13783.710606] LustreError: Skipped 1 previous similar message [13783.851722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13785.826855] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13788.124350] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13788.940460] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:34738 to 0x280000400:34753) [13788.940489] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:34823 to 0x2c0000400:34849) [13788.999836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:44674 to 0x280000403:44705) [13789.001232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:43896 to 0x2c0000403:43937) [13796.439110] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 19:42:52 (1746142972) [13797.467128] LustreError: 12735:0:(llog_osd.c:1480:llog_osd_open()) cfs_race id 15e sleeping [13799.325035] Lustre: Failing over lustre-MDT0000 [13799.436415] LustreError: 14875:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13799.448185] LustreError: 12735:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e awake: rc=3023 [13800.459481] LustreError: 14875:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13800.487867] Lustre: server umount lustre-MDT0000 complete [13804.965035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13805.045352] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13805.095848] LustreError: 15479:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13806.220866] LustreError: 15478:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13807.228302] LustreError: 15478:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13809.247980] LustreError: 15478:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13809.253536] LustreError: 15478:0:(llog_osd.c:1480:llog_osd_open()) Skipped 1 previous similar message [13813.315450] LustreError: 15477:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13813.322247] LustreError: 15477:0:(llog_osd.c:1480:llog_osd_open()) Skipped 3 previous similar messages [13817.365128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13818.370550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:43896 to 0x2c0000403:43969) [13818.371625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:44674 to 0x280000403:44737) [13818.891990] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug all all [13822.046991] LustreError: 15478:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13822.052856] LustreError: 15478:0:(llog_osd.c:1480:llog_osd_open()) Skipped 5 previous similar messages [13830.068116] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 19:43:25 (1746143005) [13834.111177] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 19:43:29 (1746143009) [13858.462999] Lustre: DEBUG MARKER: == sanity test 300c: chown [14053.370961] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 19:47:08 (1746143228) [14058.340654] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 19:47:14 (1746143234) [14063.348916] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 19:47:18 (1746143238) [14068.661600] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 19:47:24 (1746143244) [14078.317311] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 19:47:34 (1746143254) [14087.309515] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 19:47:42 (1746143262) [14125.364673] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 19:48:20 (1746143300) [14129.324993] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 19:48:25 (1746143305) [14133.483123] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 19:48:29 (1746143309) [14138.487271] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 19:48:34 (1746143314) [14139.235346] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [14140.404055] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 19:48:35 (1746143315) [14149.822146] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [14150.877315] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 19:48:46 (1746143326) [14151.515384] Lustre: *** cfs_fail_loc=1704, val=0*** [14151.517059] Lustre: 12435:0:(out_handler.c:911:out_tx_end()) lustre-MDT0001-osd: error during execution of #0 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:670: rc = -28 [14151.548250] LustreError: 12981:0:(out_handler.c:921:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:425: rc = -524 [14154.979920] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 19:48:50 (1746143330) [14159.124248] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 19:48:54 (1746143334) [14163.464061] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 19:48:58 (1746143338) [14167.267375] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 19:49:02 (1746143342) [14173.933654] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 19:49:09 (1746143349) [14179.214455] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [14182.937727] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [14186.324324] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 19:49:21 (1746143361) [14284.240106] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 19:50:59 (1746143459) [14352.469089] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 19:52:08 (1746143528) [14357.674919] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 19:52:13 (1746143533) [14358.473381] Lustre: 1450:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 3793 > trans_max 3200 [14358.478611] Lustre: 1450:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 6/24/0, destroy: 6/24/0 [14358.485507] Lustre: 1450:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 31/2640/0 [14358.489241] Lustre: 1450:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 0/0/0 [14358.494432] Lustre: 1450:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 47/938/0, delete: 32/50/0 [14358.498500] Lustre: 1450:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 30/30/0, ref_del: 28/28/0 [14358.502243] CPU: 1 PID: 1450 Comm: mdt_io00_001 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [14358.507732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [14358.511315] Call Trace: [14358.513844] [] dump_stack+0x19/0x1b [14358.516065] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [14358.518320] [] ? __llog_ctxt_put+0xd3/0x140 [obdclass] [14358.523071] [] top_trans_start+0x351/0xab0 [ptlrpc] [14358.525832] [] lod_trans_start+0x7f/0x2e0 [lod] [14358.528033] [] mdd_trans_start+0x14/0x20 [mdd] [14358.530616] [] mdd_migrate_object+0x12ee/0x2470 [mdd] [14358.533134] [] ? _raw_spin_unlock+0xe/0x20 [14358.535527] [] ? __mdt_stripe_get+0xe1/0x4e0 [mdt] [14358.539333] [] mdd_migrate+0x28/0x30 [mdd] [14358.541470] [] mdt_reint_migrate+0x1b9f/0x24b0 [mdt] [14358.544208] [] mdt_reint_rec+0x87/0x240 [mdt] [14358.546872] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [14358.550315] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [14358.552824] [] mdt_reint+0x67/0x150 [mdt] [14358.554999] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [14358.558298] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [14358.562417] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [14358.565283] [] ? put_prev_entity+0x31/0x400 [14358.568363] [] ? do_raw_spin_unlock+0x49/0x90 [14358.573338] [] ? _raw_spin_unlock_irq+0xe/0x30 [14358.576814] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [14358.580291] [] kthread+0xe4/0xf0 [14358.582872] [] ? kthread_create_on_node+0x140/0x140 [14358.585211] [] ret_from_fork_nospec_begin+0x7/0x21 [14358.589379] [] ? kthread_create_on_node+0x140/0x140 [14362.827838] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 19:52:18 (1746143538) [14367.633925] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 19:52:23 (1746143543) [14368.281242] Lustre: DEBUG MARKER: SKIP: sanity test_300ui 1 MDT only [14369.154283] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 19:52:24 (1746143544) [14373.090918] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 19:52:28 (1746143548) [14377.394798] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 19:52:32 (1746143552) [14381.277011] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 19:52:36 (1746143556) [14382.151339] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [14383.104282] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 19:52:38 (1746143558) [14411.629564] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 19:53:07 (1746143587) [14412.396834] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [14413.376316] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 19:53:08 (1746143588) [14414.024104] Lustre: *** cfs_fail_loc=720, val=0*** [14414.026102] LustreError: 12431:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [14414.030246] LustreError: 12431:0:(ofd_objects.c:1014:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [14418.368729] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 19:53:14 (1746143594) [14418.801962] Lustre: *** cfs_fail_loc=720, val=0*** [14418.804011] LustreError: 21580:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [14418.805222] LustreError: 3265:0:(ofd_objects.c:1081:ofd_destroy()) lustre-OST0001 failed to stop transaction: -5 [14418.805297] LustreError: 3265:0:(ofd_dev.c:1765:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x2c0000403:0xb0f3:0x0]: -5 [14418.821946] LustreError: 21580:0:(osd_handler.c:2213:osd_trans_stop()) Skipped 2 previous similar messages [14421.464719] LustreError: 15488:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0000-osc-MDT0000: can't cancel record: rc = -2 [14424.471665] LustreError: 15490:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel record: rc = -2 [14424.476325] LustreError: 15490:0:(osp_sync.c:1155:osp_sync_process_committed()) Skipped 1 previous similar message [14429.733951] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 19:53:25 (1746143605) [14437.147338] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 19:53:32 (1746143612) [14441.745766] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 19:53:37 (1746143617) [14445.707896] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 19:53:41 (1746143621) [14448.681711] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 19:53:44 (1746143624) [14453.941985] Lustre: 1450:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200005224:0x142:0x0]/xxx is open, migrate only dentry [14456.822746] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 19:53:52 (1746143632) [14457.414697] LNet: *** cfs_fail_loc=e001, val=100*** [14457.416775] LNet: 2322:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1830956661085184 with NID mismatch 0@lo accepted (same peer ffff88012e9b6600) [14457.929086] LNet: *** cfs_fail_loc=e001, val=100*** [14457.931040] LNet: Skipped 60 previous similar messages [14457.932519] LNet: 8427:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.203.18@tcp matched 1830956661093632 with NID mismatch 192.168.203.18@tcp accepted (same peer ffff88008ef3a800) [14457.938613] LNet: 8427:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 60 previous similar messages [14612.091777] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 19:56:27 (1746143787) [14624.852059] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 19:56:40 (1746143800) [14627.401507] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 19:56:43 (1746143803) [14697.592986] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 19:57:53 (1746143873) [14730.770614] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 19:58:26 (1746143906) [14746.073787] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 19:58:41 (1746143921) [14749.250339] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 19:58:44 (1746143924) [14756.138787] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 19:58:51 (1746143931) [14757.166859] LustreError: 2728:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14757.178117] LustreError: 2728:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 5 previous similar messages [14759.178564] LustreError: 668:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14759.183411] LustreError: 668:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 2 previous similar messages [14759.512538] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14761.523574] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14761.526942] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 2 previous similar messages [14761.584273] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14763.593574] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14763.639610] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14765.644704] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14767.726431] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14767.730014] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14769.737578] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14769.740613] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14776.168161] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14776.176585] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 3 previous similar messages [14776.787136] LustreError: 31160:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout interrupted [14779.879936] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [14792.572626] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 19:59:28 (1746143968) [14797.193647] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 19:59:32 (1746143972) [14809.488699] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 19:59:44 (1746143984) [14825.370277] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7196000 > 600000 skipping out-of-space test on OST0 [14826.348530] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 20:00:01 (1746144001) [14831.185112] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7176088 > 600000 skipping out-of-space test on OST0 [14847.309047] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 20:00:23 (1746144023) [14847.875186] Lustre: *** cfs_fail_loc=20e, val=1*** [14847.878136] Lustre: Skipped 2 previous similar messages [14848.900812] Lustre: *** cfs_fail_loc=20e, val=1*** [14848.907135] Lustre: Skipped 3 previous similar messages [14850.931348] Lustre: *** cfs_fail_loc=20e, val=1*** [14850.932955] Lustre: Skipped 1 previous similar message [14853.960534] Lustre: *** cfs_fail_loc=20e, val=1*** [14853.963794] Lustre: Skipped 5 previous similar messages [14857.980359] Lustre: *** cfs_fail_loc=20e, val=1*** [14857.982089] Lustre: Skipped 1 previous similar message [14869.013194] Lustre: *** cfs_fail_loc=20e, val=1*** [14869.015136] Lustre: Skipped 7 previous similar messages [14893.048576] Lustre: *** cfs_fail_loc=20e, val=1*** [14893.051448] Lustre: Skipped 11 previous similar messages [14925.901411] Lustre: *** cfs_fail_loc=20f, val=1*** [14925.904853] Lustre: Skipped 35 previous similar messages [14996.659685] Lustre: *** cfs_fail_loc=20e, val=2*** [14996.664358] Lustre: Skipped 54 previous similar messages [15078.325957] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 20:04:13 (1746144253) [15090.978577] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 20:04:26 (1746144266) [15091.881526] Lustre: DEBUG MARKER: sanity test_398o: @@@@@@ FAIL: bad KMS