[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490380635 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] BRK [0x115201000, 0x115201fff] PGTABLE [ 0.000000] BRK [0x115202000, 0x115202fff] PGTABLE [ 0.000000] BRK [0x115203000, 0x115203fff] PGTABLE [ 0.000000] BRK [0x115204000, 0x115204fff] PGTABLE [ 0.000000] BRK [0x115205000, 0x115205fff] PGTABLE [ 0.000000] BRK [0x115206000, 0x115206fff] PGTABLE [ 0.000000] BRK [0x115207000, 0x115207fff] PGTABLE [ 0.000000] BRK [0x115208000, 0x115208fff] PGTABLE [ 0.000000] BRK [0x115209000, 0x115209fff] PGTABLE [ 0.000000] BRK [0x11520a000, 0x11520afff] PGTABLE [ 0.000000] BRK [0x11520b000, 0x11520bfff] PGTABLE [ 0.000000] BRK [0x11520c000, 0x11520cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] On node 0 totalpages: 1076588 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:63 [ 0.000000] Normal zone: 4536 pages used for memmap [ 0.000000] Normal zone: 290304 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s221184 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001000] hpet clockevent registered [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002370] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007389] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.011122] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.021700] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025610] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029141] *** VALIDATE tmpfs *** [ 0.031083] *** VALIDATE proc *** [ 0.032161] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.034342] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037041] Spectre V2 : User space: Vulnerable [ 0.038000] Speculative Store Bypass: Vulnerable [ 0.040586] debug: unmapping init [mem 0xffffffffa9c59000-0xffffffffa9c60fff] [ 0.042279] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043647] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048011] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.049690] ... event mask: 000000070000000f [ 0.050354] rcu: Hierarchical SRCU implementation. [ 0.052492] smp: Bringing up secondary CPUs ... [ 0.053799] x86: Booting SMP configuration: [ 0.054026] .... node #0, CPUs: #1 #2 #3 [ 0.062135] smp: Brought up 1 node, 4 CPUs [ 0.064021] smpboot: Max logical packages: 1 [ 0.065037] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125209] node 0 deferred pages initialised in 44ms [ 0.139559] devtmpfs: initialized [ 0.140305] x86/mm: Memory block size: 128MB [ 0.144984] gcov: version magic: 0x41383552 [ 0.148086] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149257] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153713] pinctrl core: initialized pinctrl subsystem [ 0.159012] ************************************************************* [ 0.163018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166000] ** ** [ 0.166000] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166000] ** ** [ 0.167020] ** This means that this kernel is built to expose internal ** [ 0.168000] ** IOMMU data structures, which may compromise security on ** [ 0.172023] ** your system. ** [ 0.174011] ** ** [ 0.178023] ** If you see this message and you are not debugging the ** [ 0.181020] ** kernel, report this immediately to your vendor! ** [ 0.184018] ** ** [ 0.187018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190020] ************************************************************* [ 0.194119] NET: Registered protocol family 16 [ 0.196495] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204031] cpuidle: using governor menu [ 0.205348] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207416] PCI: Using configuration type 1 for base access [ 0.208120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217250] cryptd: max_cpu_qlen set to 1000 [ 0.219009] ACPI: Added _OSI(Module Device) [ 0.220014] ACPI: Added _OSI(Processor Device) [ 0.221032] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.230333] ACPI: Interpreter enabled [ 0.231064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232008] ACPI: Using IOAPIC for interrupt routing [ 0.233083] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244260] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.252108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257510] acpiphp: Slot [2] registered [ 0.259136] acpiphp: Slot [5] registered [ 0.261139] acpiphp: Slot [6] registered [ 0.262121] acpiphp: Slot [7] registered [ 0.263149] acpiphp: Slot [8] registered [ 0.264119] acpiphp: Slot [9] registered [ 0.265106] acpiphp: Slot [10] registered [ 0.266113] acpiphp: Slot [3] registered [ 0.268085] acpiphp: Slot [4] registered [ 0.269104] acpiphp: Slot [11] registered [ 0.270094] acpiphp: Slot [12] registered [ 0.271138] acpiphp: Slot [13] registered [ 0.273092] acpiphp: Slot [14] registered [ 0.274103] acpiphp: Slot [15] registered [ 0.275095] acpiphp: Slot [16] registered [ 0.277101] acpiphp: Slot [17] registered [ 0.278082] acpiphp: Slot [18] registered [ 0.279153] acpiphp: Slot [19] registered [ 0.280058] acpiphp: Slot [20] registered [ 0.282075] acpiphp: Slot [21] registered [ 0.283091] acpiphp: Slot [22] registered [ 0.284096] acpiphp: Slot [23] registered [ 0.286101] acpiphp: Slot [24] registered [ 0.287138] acpiphp: Slot [25] registered [ 0.288101] acpiphp: Slot [26] registered [ 0.289096] acpiphp: Slot [27] registered [ 0.291090] acpiphp: Slot [28] registered [ 0.292098] acpiphp: Slot [29] registered [ 0.293099] acpiphp: Slot [30] registered [ 0.295098] acpiphp: Slot [31] registered [ 0.296060] PCI host bridge to bus 0000:00 [ 0.297017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313141] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316095] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.326019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333177] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.336020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.344796] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348020] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.355000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.359000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.365017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373803] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385068] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.399018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.414634] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.422025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.450025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.461243] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.466020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.487025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.496470] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.503024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.511027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.526019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.533846] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.536023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.545019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.559027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.569000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.570000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.570000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.579022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.592000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.594401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.598616] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.601655] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.604405] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.614000] iommu: Default domain type: Passthrough [ 0.629966] SCSI subsystem initialized [ 0.631213] ACPI: bus type USB registered [ 0.632110] usbcore: registered new interface driver usbfs [ 0.634093] usbcore: registered new interface driver hub [ 0.636108] usbcore: registered new device driver usb [ 0.639249] pps_core: LinuxPPS API ver. 1 registered [ 0.640015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.644090] PTP clock support registered [ 0.645151] EDAC MC: Ver: 3.0.0 [ 0.649671] PCI: Using ACPI for IRQ routing [ 0.650013] PCI: pci_cache_line_size set to 64 bytes [ 0.650322] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.650340] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.650344] e820: reserve RAM buffer [mem 0x146e00000-0x147ffffff] [ 0.650997] NetLabel: Initializing [ 0.651829] NetLabel: domain hash size = 128 [ 0.653019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.654079] NetLabel: unlabeled traffic allowed by default [ 0.656140] vgaarb: loaded [ 0.657294] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.658014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.662571] clocksource: Switched to clocksource kvm-clock [ 0.791391] VFS: Disk quotas dquot_6.6.0 [ 0.794220] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802887] *** VALIDATE ramfs *** [ 0.805163] *** VALIDATE hugetlbfs *** [ 0.808456] pnp: PnP ACPI init [ 0.810489] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.810611] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.810638] pnp 00:02: [dma 2] [ 0.810678] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.810748] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.810813] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.810919] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.811852] pnp: PnP ACPI: found 6 devices [ 0.831660] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.836076] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.838626] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.841283] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.844260] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.847432] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.850991] NET: Registered protocol family 2 [ 0.854579] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.861505] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.866203] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.872942] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.877762] TCP: Hash tables configured (established 65536 bind 65536) [ 0.881836] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.886451] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.890656] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894586] NET: Registered protocol family 1 [ 0.898889] RPC: Registered named UNIX socket transport module. [ 0.901506] RPC: Registered udp transport module. [ 0.903430] RPC: Registered tcp transport module. [ 0.905706] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.908563] NET: Registered protocol family 44 [ 0.910841] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.913694] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916213] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.919278] PCI: CLS 0 bytes, default 64 [ 0.920799] Unpacking initramfs... [ 2.891396] debug: unmapping init [mem 0xffff8cf77cc54000-0xffff8cf77ffbffff] [ 2.895388] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.897654] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.901083] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.733653] Initialise system trusted keyrings [ 3.735529] Key type blacklist registered [ 3.737718] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.749226] zbud: loaded [ 3.756520] *** VALIDATE nfs *** [ 3.759207] *** VALIDATE nfs4 *** [ 3.790430] pstore: using deflate compression [ 3.795436] Platform Keyring initialized [ 3.965148] NET: Registered protocol family 38 [ 3.981234] Key type asymmetric registered [ 3.982857] Asymmetric key parser 'x509' registered [ 3.984218] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.988892] io scheduler mq-deadline registered [ 3.990299] io scheduler kyber registered [ 3.991907] io scheduler bfq registered [ 4.047909] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.052353] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.058670] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.066135] ACPI: Power Button [PWRF] [ 4.072605] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.078745] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.091150] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.105898] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.144713] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.186806] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.217660] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.222751] Non-volatile memory driver v1.3 [ 4.224176] Linux agpgart interface v0.103 [ 4.260547] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.263612] vda: detected capacity change from 0 to 68624384 [ 4.279161] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.282391] vdb: detected capacity change from 0 to 1073741824 [ 4.302252] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.304834] vdc: detected capacity change from 0 to 2621440000 [ 4.324664] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.327162] vdd: detected capacity change from 0 to 2621440000 [ 4.350647] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.353128] vde: detected capacity change from 0 to 4294967296 [ 4.366996] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.369539] vdf: detected capacity change from 0 to 4294967296 [ 4.375836] libphy: Fixed MDIO Bus: probed [ 4.382116] usbcore: registered new interface driver usbserial_generic [ 4.383914] usbserial: USB Serial support registered for generic [ 4.386146] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.390589] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.392164] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.394354] mousedev: PS/2 mouse device common for all mice [ 4.398372] rtc_cmos 00:05: RTC can wake from S4 [ 4.401599] rtc_cmos 00:05: registered as rtc0 [ 4.403012] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.407332] intel_pstate: CPU model not supported [ 4.410963] hid: raw HID events driver (C) Jiri Kosina [ 4.411639] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.414882] usbcore: registered new interface driver usbhid [ 4.414889] usbhid: USB HID core driver [ 4.415016] drop_monitor: Initializing network drop monitor service [ 4.415158] Initializing XFRM netlink socket [ 4.416864] NET: Registered protocol family 10 [ 4.422945] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.425242] Segment Routing with IPv6 [ 4.428468] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.431163] NET: Registered protocol family 17 [ 4.432317] mpls_gso: MPLS GSO support [ 4.442454] start plist test [ 4.448634] end plist test [ 4.451362] RAS: Correctable Errors collector initialized. [ 4.453161] AVX version of gcm_enc/dec engaged. [ 4.454678] AES CTR mode by8 optimization enabled [ 4.532335] sched_clock: Marking stable (4532215270, 0)->(5797749703, -1265534433) [ 4.535414] registered taskstats version 1 [ 4.537326] Loading compiled-in X.509 certificates [ 4.538819] zswap: loaded using pool lzo/zbud [ 4.567837] Key type big_key registered [ 4.580593] Key type encrypted registered [ 4.581883] ima: No TPM chip found, activating TPM-bypass! [ 4.583499] ima: Allocated hash algorithm: sha1 [ 4.584730] ima: No architecture policies found [ 4.586081] evm: Initialising EVM extended attributes: [ 4.587595] evm: security.selinux [ 4.588569] evm: security.ima [ 4.589448] evm: security.capability [ 4.590422] evm: HMAC attrs: 0x1 [ 4.592598] rtc_cmos 00:05: setting system clock to 2026-01-02 00:46:43 UTC (1767314803) [ 4.599520] debug: unmapping init [mem 0xffffffffaac03000-0xffffffffaadfffff] [ 4.602264] debug: unmapping init [mem 0xffffffffa9982000-0xffffffffa9c58fff] [ 4.614218] Write protecting the kernel read-only data: 28672k [ 4.618424] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 4.620720] debug: unmapping init [mem 0xffffffffa8914000-0xffffffffa89fffff] [ 4.753249] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.769671] systemd[1]: Detected virtualization kvm. [ 4.809075] systemd[1]: Detected architecture x86-64. [ 4.812666] systemd[1]: Running in initial RAM disk. [ 4.946731] systemd[1]: No hostname configured. [ 4.948896] systemd[1]: Set hostname to . [ 4.951818] random: systemd: uninitialized urandom read (16 bytes read) [ 4.954830] systemd[1]: Initializing machine ID from random generator. [ 5.057561] random: ln: uninitialized urandom read (6 bytes read) [ 5.661711] random: systemd: uninitialized urandom read (16 bytes read) [ 5.666363] systemd[1]: Listening on Journal Socket. [ 5.676335] systemd[1]: Listening on udev Kernel Socket. [ 5.696722] systemd[1]: Started Memstrack Anylazing Service. [ 5.973957] urandom_read: 4 callbacks suppressed [ 5.973965] random: systemd: uninitialized urandom read (16 bytes read) [ 6.054726] random: systemd: uninitialized urandom read (16 bytes read) [ 6.120218] random: systemd: uninitialized urandom read (16 bytes read) [ 7.405423] device-mapper: uevent: version 1.0.3 [ 7.407546] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 9.086956] virtio_net virtio0 ens2: renamed from eth0 [ 9.135164] random: fast init done [ 9.313106] libata version 3.00 loaded. [ 9.526841] ata_piix 0000:00:01.1: version 2.13 [ 9.547496] scsi host0: ata_piix [ 9.893981] scsi host1: ata_piix [ 9.895607] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.900012] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.166541] random: crng init done [ 18.433915] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 20.378159] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.127900] SELinux: Disabled at runtime. [ 21.301832] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.325770] systemd[1]: Detected virtualization kvm. [ 21.329770] systemd[1]: Detected architecture x86-64. [ 22.736464] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.741877] systemd[1]: Stopped Switch Root. [ 22.758313] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.769655] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.779359] systemd[1]: Stopped Journal Service. [ 22.796786] systemd[1]: Starting Journal Service... [ 22.810875] systemd[1]: Mounting Huge Pages File System... [ 23.017096] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 24.158670] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 26.124950] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.025587] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.218907] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.294749] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 28.294770] EDAC sbridge: Ver: 1.1.2 [ 31.755747] Key type dns_resolver registered [ 32.437787] NFS: Registering the id_resolver key type [ 32.443288] Key type id_resolver registered [ 32.445766] Key type id_legacy registered [ 69.966218] libcfs: loading out-of-tree module taints kernel. [ 69.997593] Key type ._llcrypt registered [ 69.998819] Key type .llcrypt registered [ 70.091368] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 89.190014] hrtimer: interrupt took 2136501 ns [ 108.208833] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 108.808538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 108.817083] alg: No test for adler32 (adler32-zlib) [ 109.869737] Lustre: Lustre: Build Version: 2.17.0_RC4_5_g36e97ed [ 110.280045] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 111.904655] Key type lgssc registered [ 112.609587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.692679] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 133.010209] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.144160] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.836940] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.995610] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.115468] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 220.342670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.501245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 220.539512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.699856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.844523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 222.080694] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 222.150586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.263373] Lustre: lustre-MDT0001: new disk, initializing [ 222.381205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.412042] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 222.424875] Lustre: 15491:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 222.438411] LustreError: 15491:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 222.535760] Lustre: 15453:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.871267] Lustre: lustre-MDT0000: new disk, initializing [ 223.119199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.170890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.237685] Lustre: 15536:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b9a637c6 x1853163998026880/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767315039 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 228.099204] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.656117] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.330217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 233.344602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 236.270863] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.336601] Lustre: 16675:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 252.021301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 252.127995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 252.380213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 252.419677] Lustre: lustre-OST0001: new disk, initializing [ 252.429465] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.438893] Lustre: Skipped 2 previous similar messages [ 252.450876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 252.505039] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.915659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 258.944706] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 259.023227] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 259.077134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000400 [ 261.659660] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.479340] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.499064] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.657260] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.159791] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.705248] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing yml_node [ 314.296345] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 319.143563] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 324.532918] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 327.886815] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 1 19:52:03 EST 2026 [ 354.061604] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 355.801894] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 376.803325] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.804242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.817755] Lustre: Skipped 2 previous similar messages [ 376.837850] Lustre: Skipped 3 previous similar messages [ 378.131677] LustreError: 22243:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 378.291579] Lustre: server umount lustre-MDT0000 complete [ 379.055909] Lustre: server umount lustre-MDT0001 complete [ 388.381995] LustreError: 19737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315187 with bad export cookie 5216859108793138585 [ 388.400240] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 388.459401] LustreError: 23190:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 388.469970] LustreError: 23190:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 388.816571] Lustre: server umount lustre-OST0000 complete [ 389.182122] Lustre: server umount lustre-OST0001 complete [ 410.239517] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 499.360687] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 522.219643] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 523.551022] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 525.546616] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 529.057346] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.300094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.374751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.637471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 551.674797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 551.683138] Lustre: Skipped 1 previous similar message [ 551.835671] Lustre: lustre-MDT0000: new disk, initializing [ 551.842615] Lustre: Skipped 1 previous similar message [ 551.949897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.952410] Lustre: Skipped 1 previous similar message [ 551.991042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 552.000715] Lustre: Skipped 1 previous similar message [ 557.518221] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 582.318439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 582.420828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 582.643777] Lustre: 31120:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 582.741468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 582.753445] Lustre: Skipped 1 previous similar message [ 582.866919] Lustre: lustre-MDT0001: new disk, initializing [ 583.029276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 583.099429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 583.109755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 583.119912] Lustre: Skipped 1 previous similar message [ 587.919756] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 596.261706] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 601.309731] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 620.717024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 620.843620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 621.264432] Lustre: lustre-OST0000: new disk, initializing [ 621.269070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 621.388643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 622.738192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 622.751232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 622.976539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 629.164407] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 639.705496] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 645.512762] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 645.892948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 651.559289] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 651.904849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 658.914945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 658.917231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.919034] LustreError: Skipped 1 previous similar message [ 658.924263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 658.956952] Lustre: Skipped 2 previous similar messages [ 660.689038] LustreError: 35092:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 660.691603] LustreError: 35092:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 661.054239] Lustre: server umount lustre-OST0000 complete [ 669.160636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.173438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 669.200847] Lustre: Skipped 1 previous similar message [ 669.220864] Lustre: Skipped 2 previous similar messages [ 672.226845] LustreError: 35341:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 672.235355] LustreError: 35341:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 672.374885] Lustre: server umount lustre-MDT0000 complete [ 677.576040] LustreError: 31112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315476 with bad export cookie 5216859108793139999 [ 677.583442] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.586318] LustreError: 31112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 677.725417] LustreError: 35592:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 677.730517] LustreError: 35592:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 677.973556] Lustre: server umount lustre-MDT0001 complete [ 683.101835] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 19:58:00 (1767315480) [ 693.669850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.082328] LustreError: 36312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.106829] LustreError: 36312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 694.151352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.236216] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 699.369963] LustreError: 36312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.497547] LustreError: 36312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.609723] LustreError: 36312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.776797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 718.835211] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 724.902658] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 729.609624] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 743.563088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 744.065623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.068309] Lustre: Skipped 1 previous similar message [ 751.349987] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 758.480955] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 773.656410] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 785.381811] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.393400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.394810] Lustre: Skipped 1 previous similar message [ 785.412310] Lustre: Skipped 1 previous similar message [ 787.329034] LustreError: 39174:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 787.334969] LustreError: 39174:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 787.658898] Lustre: server umount lustre-MDT0000 complete [ 788.456718] Lustre: server umount lustre-MDT0001 complete [ 793.152038] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315592 with bad export cookie 5216859108793140881 [ 793.153351] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 793.160811] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 793.516674] Lustre: server umount lustre-OST0000 complete [ 893.932703] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 914.467382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 914.639933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.058258] LustreError: 45004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.179427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.438910] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 922.225590] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 929.507479] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 929.833276] Lustre: 46110:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 944.928482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 952.306602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 952.461133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 952.702175] Lustre: lustre-OST0001: new disk, initializing [ 952.724429] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 954.346823] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 954.864109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 954.871862] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 955.063581] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 961.401326] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 973.054539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 977.731937] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 997.349220] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.354050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.371307] Lustre: Skipped 1 previous similar message [ 997.375151] Lustre: Skipped 2 previous similar messages [ 1003.166383] LustreError: 49365:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1003.179253] LustreError: 49365:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1003.330686] Lustre: server umount lustre-MDT0001 complete [ 1003.963351] Lustre: server umount lustre-MDT0000 complete [ 1008.313112] LustreError: 44990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315807 with bad export cookie 5216859108793141742 [ 1008.338803] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1008.438837] Lustre: server umount lustre-OST0000 complete [ 1008.688163] Lustre: server umount lustre-OST0001 complete [ 1102.794393] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1122.775588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.792031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1123.378111] LustreError: 55188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.523980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.527646] Lustre: Skipped 3 previous similar messages [ 1128.491787] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1129.365411] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1136.493282] Lustre: 56261:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1150.921426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1154.411821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1159.687058] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1162.501965] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1177.690598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1181.962631] Lustre: 58736:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1181.975313] Lustre: 58736:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1199.075950] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.090725] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1199.100702] Lustre: Skipped 7 previous similar messages [ 1199.102427] Lustre: Skipped 3 previous similar messages [ 1201.709568] LustreError: 59112:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1201.725074] LustreError: 59112:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1201.919377] Lustre: server umount lustre-MDT0000 complete [ 1202.119309] Lustre: server umount lustre-MDT0001 complete [ 1207.189156] LustreError: 57417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316006 with bad export cookie 5216859108793143030 [ 1207.200811] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.204457] LustreError: 57417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1207.355685] Lustre: server umount lustre-OST0000 complete [ 1207.797863] Lustre: server umount lustre-OST0001 complete [ 1221.826222] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 20:06:58 (1767316018) [ 1320.305219] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1341.358124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1341.540308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.809712] LustreError: 66002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.105807] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1347.440945] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1354.076906] Lustre: 67078:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1365.798317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1369.455400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1373.371186] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1377.243123] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1386.760291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1390.685573] Lustre: 69546:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1390.690337] Lustre: 69546:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1405.927402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.938462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1405.947724] Lustre: Skipped 3 previous similar messages [ 1405.956831] Lustre: Skipped 3 previous similar messages [ 1407.821933] LustreError: 69872:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1407.830520] LustreError: 69872:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1408.137979] Lustre: server umount lustre-MDT0000 complete [ 1408.276430] Lustre: server umount lustre-MDT0001 complete [ 1411.940106] LustreError: 65984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316210 with bad export cookie 5216859108793144129 [ 1411.953659] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1412.171838] Lustre: server umount lustre-OST0001 complete [ 1412.277142] Lustre: server umount lustre-OST0000 complete [ 1424.538484] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 20:10:21 (1767316221) [ 1502.582899] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1520.667987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1520.680498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.092739] LustreError: 75859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.115317] LustreError: 75859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1521.224524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.232403] Lustre: Skipped 7 previous similar messages [ 1526.568026] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1526.749222] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1532.517067] Lustre: 76937:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1544.840216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1549.249320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1552.974429] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1556.173316] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1564.096661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1574.203062] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1574.213668] Lustre: Skipped 1 previous similar message [ 1579.849727] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.42@tcp (stopping) [ 1580.516576] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.550077] Lustre: Skipped 4 previous similar messages [ 1580.701240] LustreError: 79666:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1580.710245] LustreError: 79666:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1580.871550] Lustre: server umount lustre-MDT0000 complete [ 1585.645140] LustreError: 75859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.665404] LustreError: 75859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1586.088657] LustreError: 75839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316384 with bad export cookie 5216859108793145228 [ 1586.098669] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.104639] LustreError: 75839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1592.795595] Lustre: server umount lustre-MDT0001 complete [ 1605.443787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.292498] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1621.940809] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1629.256031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1634.802553] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1634.807578] Lustre: Skipped 1 previous similar message [ 1635.418501] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1641.689643] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1643.995271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1647.936992] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1649.652582] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1660.387940] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.396133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1660.417771] Lustre: Skipped 5 previous similar messages [ 1660.437467] Lustre: Skipped 12 previous similar messages [ 1662.056176] Lustre: server umount lustre-MDT0001 complete [ 1662.172283] Lustre: server umount lustre-MDT0000 complete [ 1665.906843] LustreError: 75840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316464 with bad export cookie 5216859108793146369 [ 1665.907923] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1665.914825] LustreError: 75840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1665.993892] Lustre: server umount lustre-OST0000 complete [ 1666.244040] Lustre: server umount lustre-OST0001 complete [ 1678.314407] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 20:14:35 (1767316475) [ 1752.491353] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1770.361497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1770.363332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.706225] LustreError: 88442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1770.729225] LustreError: 88442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 1775.022349] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1775.674228] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1781.051107] Lustre: 89517:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1781.073957] Lustre: 89517:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1793.557064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1796.451780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1800.908889] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1803.838221] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1812.834300] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1822.673147] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1839.588197] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.601657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.606593] Lustre: Skipped 3 previous similar messages [ 1839.623372] Lustre: Skipped 3 previous similar messages [ 1852.384445] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1852.394403] Lustre: Skipped 1 previous similar message [ 1852.513736] LustreError: 92364:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1852.526357] LustreError: 92364:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 1852.687307] Lustre: server umount lustre-MDT0001 complete [ 1852.845739] Lustre: server umount lustre-MDT0000 complete [ 1857.408061] LustreError: 88423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316656 with bad export cookie 5216859108793147188 [ 1857.431654] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1857.538532] Lustre: server umount lustre-OST0001 complete [ 1857.886418] Lustre: server umount lustre-OST0000 complete [ 1872.640467] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 20:17:49 (1767316669) [ 1876.743213] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1878.837696] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 20:17:55 (1767316675) [ 1904.176169] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 1945.404669] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1961.959120] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1966.919497] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1983.421909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1983.495940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.774428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1983.797983] Lustre: Skipped 1 previous similar message [ 1983.868581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1983.998943] Lustre: lustre-MDT0000: new disk, initializing [ 1984.083217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1988.200351] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1995.787209] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2008.835607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2008.904624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2009.228509] Lustre: lustre-OST0000: new disk, initializing [ 2009.236690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2009.252073] Lustre: Skipped 1 previous similar message [ 2010.995497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2011.004452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2011.030111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2016.001524] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2023.904525] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2026.290188] Lustre: DEBUG MARKER: create test files [ 2061.991337] Lustre: server umount lustre-OST0000 complete [ 2067.475587] Lustre: server umount lustre-MDT0000 complete [ 2082.377844] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2085.512649] Key type lgssc unregistered [ 2085.776903] LNet: 102315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.789107] LNetError: 102315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.814929] LNet: Removed LNI 192.168.202.142@tcp [ 2086.820146] Key type .llcrypt unregistered [ 2086.822396] Key type ._llcrypt unregistered [ 2155.656678] Key type ._llcrypt registered [ 2155.659965] Key type .llcrypt registered [ 2155.803188] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2157.237595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.402548] alg: No test for adler32 (adler32-zlib) [ 2158.501239] Lustre: Lustre: Build Version: 2.17.0_RC4_5_g36e97ed [ 2158.806323] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2160.512175] Key type lgssc registered [ 2161.591910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.464702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2168.486694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.958522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.563157] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2180.669763] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2191.968568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2192.328298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.430021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 2194.538217] Lustre: 107115:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 2194.550875] Lustre: 107115:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 2194.559307] Lustre: 107115:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 2194.567262] Lustre: 107115:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 2194.581768] Lustre: 107115:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 2194.602884] Lustre: 107115:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2198.797387] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2204.966805] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2210.830503] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 2216.571990] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 2225.123762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2225.129347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.146494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2228.194147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2229.327162] LustreError: 107862:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2229.448759] Lustre: server umount lustre-OST0000 complete [ 2234.296060] LustreError: 108111:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2234.306218] LustreError: 108111:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2234.515846] Lustre: server umount lustre-MDT0000 complete [ 2248.759457] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2251.331101] Key type lgssc unregistered [ 2251.616715] LNet: 108789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.625279] LNetError: 108789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.638516] LNet: Removed LNI 192.168.202.142@tcp [ 2252.459325] Key type .llcrypt unregistered [ 2252.460132] Key type ._llcrypt unregistered [ 2319.825654] Key type ._llcrypt registered [ 2319.827741] Key type .llcrypt registered [ 2319.917931] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2321.027366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.050684] alg: No test for adler32 (adler32-zlib) [ 2322.106182] Lustre: Lustre: Build Version: 2.17.0_RC4_5_g36e97ed [ 2322.367649] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2324.105704] Key type lgssc registered [ 2325.190902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2332.236602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2332.249705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.701987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.444200] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2344.687785] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2354.787891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2355.027662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.141880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 2358.278050] Lustre: 113593:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 2358.290851] Lustre: 113593:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 2358.295110] Lustre: 113593:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 2358.307366] Lustre: 113593:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 2358.312925] Lustre: 113593:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 2358.326744] Lustre: 113593:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2361.488982] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2367.413749] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2373.238204] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 2414.564673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.596108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.970158] LustreError: 114309:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2417.114091] Lustre: server umount lustre-OST0000 complete [ 2423.353844] LustreError: 114558:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2423.358759] LustreError: 114558:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2423.584357] Lustre: server umount lustre-MDT0000 complete [ 2441.465887] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2444.657315] Key type lgssc unregistered [ 2444.999874] LNet: 115235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.010698] LNetError: 115235:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.031445] LNet: Removed LNI 192.168.202.142@tcp [ 2446.098335] Key type .llcrypt unregistered [ 2446.100251] Key type ._llcrypt unregistered [ 2480.739982] Key type ._llcrypt registered [ 2480.742393] Key type .llcrypt registered [ 2480.873869] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 2550.987394] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2552.157507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.289537] alg: No test for adler32 (adler32-zlib) [ 2553.380399] Lustre: Lustre: Build Version: 2.17.0_RC4_5_g36e97ed [ 2553.677371] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2555.312173] Key type lgssc registered [ 2556.505870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2580.256412] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2580.873058] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2582.723183] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2585.880751] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2604.223202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2604.322993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2604.347106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.803530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2605.865805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2605.956427] Lustre: lustre-MDT0000: new disk, initializing [ 2606.157122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.177556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2610.529581] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2627.715595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2627.811561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2627.921812] Lustre: 124603:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2627.968687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2627.972807] Lustre: Skipped 1 previous similar message [ 2628.035565] Lustre: lustre-MDT0001: new disk, initializing [ 2628.133674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2628.175806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2628.191228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2632.785706] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2639.845604] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2644.313587] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2657.566965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2657.636821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2657.863410] Lustre: lustre-OST0000: new disk, initializing [ 2657.866542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2657.940747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.950030] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2664.068391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2664.085701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2664.235136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2673.069320] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2679.405500] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2679.684985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2685.038458] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2685.312799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2690.028233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.038674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2690.047926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.062192] Lustre: Skipped 1 previous similar message [ 2693.500673] LustreError: 128238:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2693.682010] Lustre: server umount lustre-OST0000 complete [ 2699.745228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2699.754190] LustreError: Skipped 1 previous similar message [ 2699.757619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.774302] Lustre: Skipped 1 previous similar message [ 2699.781702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.229983] LustreError: 128489:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2704.233820] LustreError: 128489:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2704.373328] Lustre: server umount lustre-MDT0000 complete [ 2709.436766] LustreError: 124595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317508 with bad export cookie 8331707638129036821 [ 2709.437956] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2709.448965] LustreError: 124595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2709.615671] LustreError: 128739:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2709.619393] LustreError: 128739:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2709.827284] Lustre: server umount lustre-MDT0001 complete [ 2719.619054] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 20:31:56 (1767317516) [ 2800.655649] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2822.697231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2861.617600] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 2932.905272] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2954.204300] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2954.890531] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2957.522800] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2959.726897] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2975.643539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2975.693866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.864704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2975.894467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2975.946983] Lustre: lustre-MDT0000: new disk, initializing [ 2975.981571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.989265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2979.297206] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2998.056071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2998.206727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2998.322584] Lustre: 142508:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2998.410200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2998.418267] Lustre: Skipped 1 previous similar message [ 2998.544883] Lustre: lustre-MDT0001: new disk, initializing [ 2998.651649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2998.703460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2998.714271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3003.057975] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3011.207744] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3015.874860] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3029.116620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.197976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3029.514467] Lustre: lustre-OST0000: new disk, initializing [ 3029.520791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3029.592906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.092338] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3036.791168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3036.808343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3036.943832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3044.570197] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3049.750938] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3050.097642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3056.108632] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3056.461477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3062.775967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3062.785322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.786109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3062.798846] Lustre: Skipped 2 previous similar messages [ 3062.808221] Lustre: Skipped 3 previous similar messages [ 3065.278693] LustreError: 146143:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3065.284765] LustreError: 146143:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3065.630389] Lustre: server umount lustre-OST0000 complete [ 3072.996463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3073.009717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3073.011545] Lustre: Skipped 1 previous similar message [ 3073.038317] Lustre: Skipped 1 previous similar message [ 3077.626982] LustreError: 146393:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3077.638606] LustreError: 146393:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3077.786356] Lustre: server umount lustre-MDT0000 complete [ 3078.119075] LustreError: 143988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.160619] LustreError: 143988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3082.217280] LustreError: 143988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.239830] LustreError: 143988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3083.930523] LustreError: 142502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317882 with bad export cookie 8331707638129037703 [ 3083.934750] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3083.946548] LustreError: 142502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3084.583220] Lustre: server umount lustre-MDT0001 complete [ 3095.268469] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 20:38:12 (1767317892) [ 3103.936572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.194564] LustreError: 147508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.264735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3108.133453] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3109.347761] LustreError: 147508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.466067] LustreError: 147508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.692105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3126.099587] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3133.380640] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3137.674556] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3150.602486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3150.962469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.975682] Lustre: Skipped 1 previous similar message [ 3157.353613] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3164.030124] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3167.849431] Lustre: 150234:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 89576949-1b5e-4bed-a897-6ba81de604e5 at adminstrative request [ 3170.789460] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.804995] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 3181.032975] Lustre: MGS: Client 89576949-1b5e-4bed-a897-6ba81de604e5 (at 0@lo) reconnecting [ 3181.037365] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x73a02bf41f1714f9 to 0x73a02bf41f171792 [ 3181.043859] Lustre: MGC192.168.202.142@tcp: Connection restored to 0@lo (at 0@lo) [ 3215.242995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3215.340693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3215.567650] Lustre: lustre-OST0001: new disk, initializing [ 3215.570902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3215.777451] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3221.681191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3221.691053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3221.803715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3222.476155] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3232.468517] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3237.366291] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3237.368215] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.379406] LustreError: Skipped 2 previous similar messages [ 3237.382413] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3237.426412] Lustre: Skipped 1 previous similar message [ 3241.408092] LustreError: 151909:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3241.411115] LustreError: 151909:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3241.593248] Lustre: server umount lustre-OST0001 complete [ 3242.484270] LustreError: 150247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.496919] LustreError: 150247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3246.564470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3246.573805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.601452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3246.610027] Lustre: Skipped 1 previous similar message [ 3251.659772] Lustre: server umount lustre-OST0000 complete [ 3262.946262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3262.957025] Lustre: Skipped 5 previous similar messages [ 3271.136184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3271.272305] LustreError: 152361:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3271.278697] LustreError: 152361:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3271.450965] Lustre: server umount lustre-MDT0000 complete [ 3273.187320] LustreError: 148414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.230612] LustreError: 148414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3276.305194] LustreError: 147488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318075 with bad export cookie 8331707638129039250 [ 3276.309978] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3276.325100] LustreError: 147488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3276.607389] Lustre: server umount lustre-MDT0001 complete [ 3286.325884] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 20:41:23 (1767318083) [ 3288.233946] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 3291.314166] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 20:41:27 (1767318087) [ 3293.539486] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 3295.569949] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 20:41:32 (1767318092) [ 3297.652817] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 3299.972237] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 20:41:36 (1767318096) [ 3301.751100] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 3303.625962] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 20:41:40 (1767318100) [ 3306.375738] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 3311.366922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3312.659540] loop: module loaded [ 3312.699183] loop0: detected capacity change from 0 to 4294967296 [ 3312.825144] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3314.760613] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 3318.650916] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 3322.666324] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 3332.110716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3337.388735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3342.657124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3359.749551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3359.832842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3359.863925] LustreError: 155719:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-28) [ 3359.876993] LustreError: 155719:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 3359.882636] LustreError: 155719:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 3359.891773] LustreError: 155719:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 3391.661486] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 3457.588120] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3478.363585] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3478.454521] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3479.496147] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3482.179975] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3497.581359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3497.628539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.830470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3497.857788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3497.930456] Lustre: lustre-MDT0000: new disk, initializing [ 3498.041806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.065783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3502.560630] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3519.831573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3519.910655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3519.988169] Lustre: 164668:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3520.302397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3524.786813] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3532.981949] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3537.278329] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3551.930290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.035519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3558.451176] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3561.530793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3561.693490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3566.365280] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3571.347723] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3571.521575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3576.368708] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3576.570557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3582.434140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.442361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3582.458447] Lustre: Skipped 3 previous similar messages [ 3582.464619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3582.464789] LustreError: Skipped 2 previous similar messages [ 3582.476831] Lustre: Skipped 3 previous similar messages [ 3584.576116] LustreError: 168300:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3584.590669] LustreError: 168300:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3584.789114] Lustre: server umount lustre-OST0000 complete [ 3595.647459] Lustre: server umount lustre-MDT0000 complete [ 3597.793955] LustreError: 164674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.803453] LustreError: 164674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3600.021294] LustreError: 164660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318398 with bad export cookie 8331707638129039852 [ 3600.030431] LustreError: 164660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3600.033345] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.298559] Lustre: server umount lustre-MDT0001 complete [ 3608.885787] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 20:46:45 (1767318405) [ 3620.069090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.942617] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3638.859234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3644.478928] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3651.206411] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3655.445958] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3667.091374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3673.891409] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3679.572807] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3684.404700] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3692.082554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3696.463976] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3696.654311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3718.102165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3718.182602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3718.303160] Lustre: lustre-OST0001: new disk, initializing [ 3718.306124] Lustre: Skipped 2 previous similar messages [ 3718.315458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3718.324306] Lustre: Skipped 3 previous similar messages [ 3724.945207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3724.949604] Lustre: Skipped 2 previous similar messages [ 3724.953319] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3725.119221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3725.339859] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3734.105325] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3737.718901] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3737.961714] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3741.690753] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3741.990439] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3764.696745] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3790.851170] Lustre: Failing over lustre-MDT0000 [ 3791.170313] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.42@tcp (stopping) [ 3791.172345] Lustre: Skipped 2 previous similar messages [ 3791.861491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.877546] Lustre: Skipped 5 previous similar messages [ 3793.243696] LustreError: 175603:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3793.250374] LustreError: 175603:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3793.448887] Lustre: server umount lustre-MDT0000 complete [ 3796.250848] LustreError: 169711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3796.264124] LustreError: 169711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3812.817139] Lustre: 121073:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318595/real 1767318595] req@000000009e4ac675 x1853166559979648/t0(0) o400->MGC192.168.202.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767318611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3812.847729] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3831.534910] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3831.538428] LDISKFS-fs (dm-0): recovery complete [ 3831.549440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.478712] LustreError: 121072:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c9b66fdb x1853166559990016/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3837.840559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 3837.846788] Lustre: Skipped 6 previous similar messages [ 3837.916595] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3842.322407] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3842.336598] Lustre: 169712:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3842.367343] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3843.058924] Lustre: 169716:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3843.069834] Lustre: 169716:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 3843.082582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3843.178057] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3843.512202] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3843.849874] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3843.865596] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 1 previous similar message [ 3844.192204] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3844.200736] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 1 previous similar message [ 3844.217910] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3844.228067] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 7 previous similar messages [ 3844.877941] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3844.890872] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 2 previous similar messages [ 3845.224195] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3845.227713] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 2 previous similar messages [ 3846.516578] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3846.530129] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 6 previous similar messages [ 3847.168945] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3847.180726] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 6 previous similar messages [ 3847.511761] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3847.521518] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 6 previous similar messages [ 3850.697973] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3850.709632] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 12 previous similar messages [ 3851.353941] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3851.366292] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 12 previous similar messages [ 3851.688106] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3851.694374] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 12 previous similar messages [ 3858.706917] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3858.720829] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 24 previous similar messages [ 3859.378707] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3859.402086] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 24 previous similar messages [ 3859.738127] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3859.740906] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 24 previous similar messages [ 3875.017340] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3875.052296] Lustre: 176979:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 50 previous similar messages [ 3875.388218] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3875.420417] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 49 previous similar messages [ 3875.752127] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3875.763505] LustreError: 176979:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 49 previous similar messages [ 3902.272168] Lustre: 176979:0:(ldlm_lib.c:2377:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3902.281499] Lustre: 176979:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3902.297684] Lustre: 176979:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 5ff87ee3-ba82-4c5d-aa12-1b56ad6f0a5e@192.168.202.42@tcp [ 3902.316663] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3902.322889] Lustre: 176979:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@0000000005659586 x1853166486363392/t0(8589934781) o101->5ff87ee3-ba82-4c5d-aa12-1b56ad6f0a5e@192.168.202.42@tcp:366/0 lens 592/0 e 0 to 0 dl 1767318711 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3902.356803] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3902.389143] Lustre: lustre-MDT0000: Denying connection for new client 5ff87ee3-ba82-4c5d-aa12-1b56ad6f0a5e (at 192.168.202.42@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3902.409294] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3902.435294] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3902.437425] Lustre: Skipped 2 previous similar messages [ 3902.451638] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3902.503663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3902.506626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 3920.365532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.378798] LustreError: Skipped 1 previous similar message [ 3920.382351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.393615] Lustre: Skipped 1 previous similar message [ 3920.399727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.408687] Lustre: Skipped 4 previous similar messages [ 3925.346524] LustreError: 177643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3925.352602] LustreError: 177643:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3925.404586] Lustre: server umount lustre-OST0000 complete [ 3929.060179] LustreError: 173965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.089188] LustreError: 173965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 53 previous similar messages [ 3942.880136] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3943.117343] Lustre: server umount lustre-OST0001 complete [ 3962.336150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3962.776487] Lustre: server umount lustre-MDT0000 complete [ 3967.042761] LustreError: 169694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318765 with bad export cookie 8331707638129091071 [ 3967.046513] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3967.054709] LustreError: 169694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3967.409426] Lustre: server umount lustre-MDT0001 complete [ 3975.947889] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 20:52:53 (1767318773) [ 3988.449312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3988.457490] Lustre: *** cfs_fail_loc=197, val=0*** [ 4004.002384] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4009.120880] LustreError: 179231:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 4014.279946] Lustre: *** cfs_fail_loc=197, val=0*** [ 4014.284679] Lustre: Skipped 65 previous similar messages [ 4041.065346] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4046.288340] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4048.116908] Lustre: server umount lustre-OST0000 complete [ 4055.205720] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 20:54:12 (1767318852) [ 4089.831893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.359894] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4102.750249] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4106.811168] LustreError: 181495:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4106.824932] LustreError: 181495:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4106.852759] LustreError: 181495:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4107.608703] Lustre: server umount lustre-MDT0000 complete [ 4121.302094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4150.847572] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 4213.958135] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 4233.214661] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4233.396639] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4234.934692] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4236.940126] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4252.876804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4252.954482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.170616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4253.203803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4253.309608] Lustre: lustre-MDT0000: new disk, initializing [ 4253.458834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4257.933931] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4274.996425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4275.041743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4275.114633] Lustre: 191985:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4275.408465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4279.747141] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4286.492901] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4289.890304] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4302.128873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4302.181169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4307.580508] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4312.769325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4314.173985] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4317.802355] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4318.014328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4321.743630] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4321.926838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4328.418050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4328.422709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.434630] LustreError: Skipped 3 previous similar messages [ 4328.436208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4328.436216] Lustre: Skipped 15 previous similar messages [ 4328.486511] Lustre: Skipped 6 previous similar messages [ 4338.144240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4338.184078] LustreError: 195616:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4338.188077] LustreError: 195616:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 4338.281789] Lustre: server umount lustre-OST0000 complete [ 4348.694790] Lustre: server umount lustre-MDT0000 complete [ 4348.898795] LustreError: 191996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.910327] LustreError: 191996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4352.681209] LustreError: 194434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319151 with bad export cookie 8331707638129101746 [ 4352.687331] LustreError: 194434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4352.689650] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.038962] Lustre: server umount lustre-MDT0001 complete [ 4360.352161] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 20:59:17 (1767319157) [ 4368.773967] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 4370.781341] Key type lgssc unregistered [ 4370.973896] LNet: 196890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4370.979283] LNetError: 196890:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4370.993244] LNet: Removed LNI 192.168.202.142@tcp [ 4371.630696] Key type .llcrypt unregistered [ 4371.634379] Key type ._llcrypt unregistered [ 4397.790784] Key type ._llcrypt registered [ 4397.795408] Key type .llcrypt registered [ 4397.912934] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 4459.289155] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 4460.448830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4460.456993] alg: No test for adler32 (adler32-zlib) [ 4461.479309] Lustre: Lustre: Build Version: 2.17.0_RC4_5_g36e97ed [ 4461.644251] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 4463.256184] Key type lgssc registered [ 4464.304914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4481.156461] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4481.181365] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4483.198269] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.172880] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.167990] Lustre: DEBUG MARKER: conf-sanity test_87: @@@@@@ FAIL: add mds1 failed [ 4504.654741] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 21:01:41 (1767319301) [ 4569.892800] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 4584.875772] Lustre: DEBUG MARKER: conf-sanity test_88: @@@@@@ FAIL: add mds1 failed [ 4589.491698] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 21:03:06 (1767319386) [ 4606.245524] Lustre: DEBUG MARKER: conf-sanity test_89: @@@@@@ FAIL: tunefs --param failover.node=192.0.2.254@tcp0 failed [ 4612.080462] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 21:03:28 (1767319408) [ 4626.285705] Lustre: DEBUG MARKER: conf-sanity test_90a: @@@@@@ FAIL: MDT start failed [ 4632.307948] Lustre: DEBUG MARKER: == conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 21:03:49 (1767319429) [ 4644.489706] Lustre: DEBUG MARKER: conf-sanity test_90b: @@@@@@ FAIL: MDT start failed [ 4650.157591] Lustre: DEBUG MARKER: == conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 21:04:07 (1767319447) [ 4662.795993] Lustre: DEBUG MARKER: conf-sanity test_90c: @@@@@@ FAIL: MDT start failed