[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 444796227 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001021] APIC: Switch to symmetric I/O mode setup [ 0.002329] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009127] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011031] SELinux: Initializing. [ 0.012057] *** VALIDATE selinux *** [ 0.020664] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025575] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027034] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030099] *** VALIDATE tmpfs *** [ 0.031428] *** VALIDATE proc *** [ 0.032231] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.035073] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042116] debug: unmapping init [mem 0xffffffff88259000-0xffffffff88260fff] [ 0.044277] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045621] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046112] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.051992] ... event mask: 000000070000000f [ 0.052306] rcu: Hierarchical SRCU implementation. [ 0.054645] smp: Bringing up secondary CPUs ... [ 0.055533] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.061183] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064009] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.131937] node 0 deferred pages initialised in 66ms [ 0.135154] devtmpfs: initialized [ 0.136230] x86/mm: Memory block size: 128MB [ 0.138751] gcov: version magic: 0x41383552 [ 0.140387] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145319] pinctrl core: initialized pinctrl subsystem [ 0.147178] [ 0.147794] ************************************************************* [ 0.150012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152009] ** ** [ 0.153008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155008] ** ** [ 0.156007] ** This means that this kernel is built to expose internal ** [ 0.158008] ** IOMMU data structures, which may compromise security on ** [ 0.159007] ** your system. ** [ 0.160007] ** ** [ 0.161007] ** If you see this message and you are not debugging the ** [ 0.163008] ** kernel, report this immediately to your vendor! ** [ 0.164010] ** ** [ 0.166012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168009] ************************************************************* [ 0.170768] NET: Registered protocol family 16 [ 0.172461] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182079] cpuidle: using governor menu [ 0.183789] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187020] PCI: Using configuration type 1 for base access [ 0.188122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198074] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201054] cryptd: max_cpu_qlen set to 1000 [ 0.203201] ACPI: Added _OSI(Module Device) [ 0.205011] ACPI: Added _OSI(Processor Device) [ 0.206000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211437] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.217366] ACPI: Interpreter enabled [ 0.218046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219015] ACPI: Using IOAPIC for interrupt routing [ 0.221119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233058] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243979] acpiphp: Slot [2] registered [ 0.245086] acpiphp: Slot [5] registered [ 0.246049] acpiphp: Slot [6] registered [ 0.247068] acpiphp: Slot [7] registered [ 0.248098] acpiphp: Slot [8] registered [ 0.248948] acpiphp: Slot [9] registered [ 0.250067] acpiphp: Slot [10] registered [ 0.251082] acpiphp: Slot [3] registered [ 0.251923] acpiphp: Slot [4] registered [ 0.252055] acpiphp: Slot [11] registered [ 0.252917] acpiphp: Slot [12] registered [ 0.254048] acpiphp: Slot [13] registered [ 0.255075] acpiphp: Slot [14] registered [ 0.256097] acpiphp: Slot [15] registered [ 0.258108] acpiphp: Slot [16] registered [ 0.259079] acpiphp: Slot [17] registered [ 0.260068] acpiphp: Slot [18] registered [ 0.261020] acpiphp: Slot [19] registered [ 0.261929] acpiphp: Slot [20] registered [ 0.263065] acpiphp: Slot [21] registered [ 0.264118] acpiphp: Slot [22] registered [ 0.266100] acpiphp: Slot [23] registered [ 0.267083] acpiphp: Slot [24] registered [ 0.268090] acpiphp: Slot [25] registered [ 0.270094] acpiphp: Slot [26] registered [ 0.271095] acpiphp: Slot [27] registered [ 0.273122] acpiphp: Slot [28] registered [ 0.274098] acpiphp: Slot [29] registered [ 0.275116] acpiphp: Slot [30] registered [ 0.276131] acpiphp: Slot [31] registered [ 0.276862] PCI host bridge to bus 0000:00 [ 0.278013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.280023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.292025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.295251] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.305018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.309403] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.313013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316890] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.318490] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.320032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.323865] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.341017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352228] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.357019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.362018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.379028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.387032] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.391016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.395013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.409016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.415138] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.420010] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.424012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.441022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.451900] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.464015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.484015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.496316] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.503017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.511016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.521013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.531248] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.536013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.540016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.554018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.563000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.567327] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.568379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.571351] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.573140] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.577050] iommu: Default domain type: Passthrough [ 0.578303] SCSI subsystem initialized [ 0.579089] ACPI: bus type USB registered [ 0.580038] usbcore: registered new interface driver usbfs [ 0.581074] usbcore: registered new interface driver hub [ 0.583074] usbcore: registered new device driver usb [ 0.584159] pps_core: LinuxPPS API ver. 1 registered [ 0.586010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.588058] PTP clock support registered [ 0.590163] EDAC MC: Ver: 3.0.0 [ 0.592167] PCI: Using ACPI for IRQ routing [ 0.594073] NetLabel: Initializing [ 0.596017] NetLabel: domain hash size = 128 [ 0.597014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.599108] NetLabel: unlabeled traffic allowed by default [ 0.601210] vgaarb: loaded [ 0.602373] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.605022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.610902] clocksource: Switched to clocksource kvm-clock [ 0.733061] VFS: Disk quotas dquot_6.6.0 [ 0.734648] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.737663] *** VALIDATE ramfs *** [ 0.739125] *** VALIDATE hugetlbfs *** [ 0.741073] pnp: PnP ACPI init [ 0.743873] pnp: PnP ACPI: found 6 devices [ 0.763832] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.767056] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.768479] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.770160] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.772504] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.774665] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.777200] NET: Registered protocol family 2 [ 0.779523] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.786035] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.789674] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.794387] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.797675] TCP: Hash tables configured (established 65536 bind 65536) [ 0.800497] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.803417] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.805778] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.807841] NET: Registered protocol family 1 [ 0.810189] RPC: Registered named UNIX socket transport module. [ 0.812164] RPC: Registered udp transport module. [ 0.813893] RPC: Registered tcp transport module. [ 0.815366] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.817550] NET: Registered protocol family 44 [ 0.819177] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.821223] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.823282] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.825711] PCI: CLS 0 bytes, default 64 [ 0.829582] Unpacking initramfs... [ 2.270797] debug: unmapping init [mem 0xffffa0347cc54000-0xffffa0347ffbffff] [ 2.276690] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.279089] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.283563] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.776477] Initialise system trusted keyrings [ 2.777610] Key type blacklist registered [ 2.779424] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.789313] zbud: loaded [ 2.793074] *** VALIDATE nfs *** [ 2.794262] *** VALIDATE nfs4 *** [ 2.795707] pstore: using deflate compression [ 2.799399] Platform Keyring initialized [ 2.904461] NET: Registered protocol family 38 [ 2.906322] Key type asymmetric registered [ 2.907857] Asymmetric key parser 'x509' registered [ 2.909593] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.912612] io scheduler mq-deadline registered [ 2.914323] io scheduler kyber registered [ 2.916242] io scheduler bfq registered [ 2.918310] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.920187] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.922138] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.924516] ACPI: Power Button [PWRF] [ 2.928641] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.933207] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.940581] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.947418] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.958700] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.984936] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.012415] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.017280] Non-volatile memory driver v1.3 [ 3.018461] Linux agpgart interface v0.103 [ 3.049591] virtio_blk virtio1: [vda] 134720 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.051699] vda: detected capacity change from 0 to 68976640 [ 3.065747] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.068171] vdb: detected capacity change from 0 to 1073741824 [ 3.080536] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.082503] vdc: detected capacity change from 0 to 2621440000 [ 3.099737] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.103054] vdd: detected capacity change from 0 to 2621440000 [ 3.128180] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.131409] vde: detected capacity change from 0 to 4294967296 [ 3.146811] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.152112] vdf: detected capacity change from 0 to 4294967296 [ 3.167698] libphy: Fixed MDIO Bus: probed [ 3.178296] usbcore: registered new interface driver usbserial_generic [ 3.181443] usbserial: USB Serial support registered for generic [ 3.184388] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.189954] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.192109] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.195513] mousedev: PS/2 mouse device common for all mice [ 3.198629] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.201839] rtc_cmos 00:05: RTC can wake from S4 [ 3.205784] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.206337] rtc_cmos 00:05: registered as rtc0 [ 3.210273] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.211206] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.212431] intel_pstate: CPU model not supported [ 3.217050] hid: raw HID events driver (C) Jiri Kosina [ 3.218453] usbcore: registered new interface driver usbhid [ 3.219806] usbhid: USB HID core driver [ 3.220892] drop_monitor: Initializing network drop monitor service [ 3.222925] Initializing XFRM netlink socket [ 3.224912] NET: Registered protocol family 10 [ 3.227313] Segment Routing with IPv6 [ 3.228366] NET: Registered protocol family 17 [ 3.229786] mpls_gso: MPLS GSO support [ 3.236156] RAS: Correctable Errors collector initialized. [ 3.238178] AVX version of gcm_enc/dec engaged. [ 3.239423] AES CTR mode by8 optimization enabled [ 3.306392] sched_clock: Marking stable (3306361551, 0)->(4095550041, -789188490) [ 3.310031] registered taskstats version 1 [ 3.311606] Loading compiled-in X.509 certificates [ 3.313250] zswap: loaded using pool lzo/zbud [ 3.332886] Key type big_key registered [ 3.346073] Key type encrypted registered [ 3.347295] ima: No TPM chip found, activating TPM-bypass! [ 3.348825] ima: Allocated hash algorithm: sha1 [ 3.349981] ima: No architecture policies found [ 3.351174] evm: Initialising EVM extended attributes: [ 3.352470] evm: security.selinux [ 3.353343] evm: security.ima [ 3.354081] evm: security.capability [ 3.354929] evm: HMAC attrs: 0x1 [ 3.356919] rtc_cmos 00:05: setting system clock to 2026-03-02 07:21:48 UTC (1772436108) [ 3.362817] debug: unmapping init [mem 0xffffffff89203000-0xffffffff893fffff] [ 3.365051] debug: unmapping init [mem 0xffffffff87f82000-0xffffffff88258fff] [ 3.373074] Write protecting the kernel read-only data: 28672k [ 3.375862] debug: unmapping init [mem 0xffffffff86603000-0xffffffff867fffff] [ 3.377837] debug: unmapping init [mem 0xffffffff86f14000-0xffffffff86ffffff] [ 3.409109] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.415376] systemd[1]: Detected virtualization kvm. [ 3.416722] systemd[1]: Detected architecture x86-64. [ 3.418012] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.440431] systemd[1]: No hostname configured. [ 3.441723] systemd[1]: Set hostname to . [ 3.443238] random: systemd: uninitialized urandom read (16 bytes read) [ 3.444960] systemd[1]: Initializing machine ID from random generator. [ 3.496855] random: ln: uninitialized urandom read (6 bytes read) [ 3.577902] random: systemd: uninitialized urandom read (16 bytes read) [ 3.579992] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.583078] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.588798] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.275611] device-mapper: uevent: version 1.0.3 [ 4.277855] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.970540] virtio_net virtio0 ens2: renamed from eth0 [ 4.985735] random: fast init done [ 5.034721] scsi host0: ata_piix [ 5.050904] scsi host1: ata_piix [ 5.052667] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.055043] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.548990] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.860246] random: crng init done [ 9.861631] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.321865] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.470762] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.705942] SELinux: Disabled at runtime. [ 11.770543] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.779326] systemd[1]: Detected virtualization kvm. [ 11.781202] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.276256] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.279868] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.283232] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.288682] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.292706] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.304102] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.311738] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ 12.367424] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.772883] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.167784] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.257166] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.341232] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.369152] EDAC sbridge: Ver: 1.1.2 [ 14.820583] Key type dns_resolver registered [ 15.120403] NFS: Registering the id_resolver key type [ 15.122335] Key type id_resolver registered [ 15.123912] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg447-server login: [ 50.297221] hrtimer: interrupt took 2730329 ns [ 63.932531] libcfs: loading out-of-tree module taints kernel. [ 63.986787] Key type ._llcrypt registered [ 63.989676] Key type .llcrypt registered [ 64.070898] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 80.631870] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 81.800228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.815641] alg: No test for adler32 (adler32-zlib) [ 83.178041] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 83.881638] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 85.671402] Key type lgssc registered [ 87.350768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.273439] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 138.743848] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 151.267476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.299879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.620237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 152.662853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 152.768286] Lustre: lustre-MDT0000: new disk, initializing [ 152.911674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.937746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 157.367863] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.154870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.274822] Lustre: 6517:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.306679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.315414] Lustre: Skipped 1 previous similar message [ 170.447714] Lustre: lustre-MDT0001: new disk, initializing [ 170.526118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.549048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.552063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.333033] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.052461] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.908254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.117769] Lustre: lustre-OST0000: new disk, initializing [ 188.121148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 188.165944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.500714] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.060720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.066633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.157492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.738310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.900077] Lustre: lustre-OST0001: new disk, initializing [ 205.902631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.972748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.734691] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.606455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.625941] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.697622] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 222.141503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.300919] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.597939] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.155834] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing yml_node [ 246.601410] Lustre: DEBUG MARKER: Client: 2.17.50.193 [ 248.890827] Lustre: DEBUG MARKER: MDS: 2.17.50.193 [ 251.037375] Lustre: DEBUG MARKER: OSS: 2.17.50.193 [ 252.339622] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 2 02:25:57 EST 2026 [ 267.865796] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 269.092799] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.688487] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.689474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.712724] Lustre: Skipped 3 previous similar messages [ 286.716291] Lustre: Skipped 2 previous similar messages [ 290.226098] Lustre: server umount lustre-MDT0000 complete [ 294.055356] LustreError: 7447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436399 with bad export cookie 3575438611846590563 [ 294.061241] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.073727] LustreError: 7447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.929537] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.930493] LustreError: 10034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.961943] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.961958] Lustre: Skipped 1 previous similar message [ 296.997845] LustreError: 10034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 299.937037] LustreError: 10034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.938711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.958818] Lustre: Skipped 2 previous similar messages [ 300.435818] Lustre: server umount lustre-MDT0001 complete [ 303.264608] Lustre: server umount lustre-OST0000 complete [ 306.462245] Lustre: server umount lustre-OST0001 complete [ 312.465530] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 319.588499] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 347.403602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.623172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 347.642257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 347.712234] Lustre: lustre-MDT0000: new disk, initializing [ 347.803806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.821631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 351.735899] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 361.057413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.207270] Lustre: 14989:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 361.294802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 361.299073] Lustre: Skipped 1 previous similar message [ 361.409105] Lustre: lustre-MDT0001: new disk, initializing [ 361.509196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 361.559698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 361.573355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 366.043704] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 373.806202] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 377.195634] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 383.291513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 383.534310] Lustre: lustre-OST0000: new disk, initializing [ 383.539346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 383.626854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.834592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 384.854517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 384.936062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 388.775536] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 395.911080] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 399.443294] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 399.602324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 403.351755] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 403.519870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 405.474185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.475756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 405.493735] Lustre: Skipped 1 previous similar message [ 405.500225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 410.595867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 410.613054] Lustre: Skipped 2 previous similar messages [ 419.296851] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 419.458965] Lustre: server umount lustre-OST0000 complete [ 422.884307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 422.889883] LustreError: Skipped 1 previous similar message [ 422.895733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.906225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 422.911854] Lustre: Skipped 2 previous similar messages [ 437.215784] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 437.458293] Lustre: server umount lustre-MDT0000 complete [ 440.868786] LustreError: 14980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436546 with bad export cookie 3575438611846592096 [ 440.887404] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.175536] Lustre: server umount lustre-MDT0001 complete [ 445.663217] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 02:29:10 (1772436550) [ 460.187548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.625976] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.651548] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 460.694672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.490255] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 465.890558] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.990787] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.370641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.190643] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 480.020418] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 483.274390] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 485.063688] Lustre: Failing over lustre-MDT0000 [ 485.294627] Lustre: server umount lustre-MDT0000 complete [ 486.370830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.374352] LustreError: 20167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.384588] Lustre: Skipped 2 previous similar messages [ 502.371860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.449850] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.703455] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 507.886456] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 507.911759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 507.950622] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 512.210551] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 516.642161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.647164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.659941] Lustre: Skipped 1 previous similar message [ 516.678854] Lustre: Skipped 7 previous similar messages [ 521.926512] Lustre: server umount lustre-MDT0000 complete [ 524.564739] LustreError: 20797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436629 with bad export cookie 3575438611846593510 [ 524.567037] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.572773] LustreError: 20797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 524.836561] Lustre: server umount lustre-MDT0001 complete [ 530.537033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.844197] LustreError: 22451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.854704] LustreError: 22451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 530.891436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.895816] Lustre: Skipped 2 previous similar messages [ 533.647950] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 539.442257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.992244] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 546.933168] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 549.463464] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 550.925212] Lustre: Failing over lustre-MDT0000 [ 551.050709] Lustre: server umount lustre-MDT0000 complete [ 553.952794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.966623] Lustre: Skipped 1 previous similar message [ 564.196377] LustreError: 22447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.216476] LustreError: 22447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 567.154639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.235440] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.067850] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 572.898814] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 572.902601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 572.908330] Lustre: Skipped 1 previous similar message [ 572.925358] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 576.194281] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 582.116859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 582.122063] Lustre: Skipped 3 previous similar messages [ 585.613593] Lustre: server umount lustre-MDT0000 complete [ 588.157488] LustreError: 22431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436693 with bad export cookie 3575438611846594448 [ 588.164625] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.169220] LustreError: 22431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 588.395973] Lustre: server umount lustre-MDT0001 complete [ 594.449667] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 02:31:39 (1772436699) [ 605.405980] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 612.559607] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 637.825026] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 645.061253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.219557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 645.236955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 645.290404] Lustre: lustre-MDT0000: new disk, initializing [ 645.357848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 647.754687] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 654.455023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.523887] Lustre: 28425:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 654.717127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 657.708551] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 663.326115] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 665.534027] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 670.081039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.346339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.354911] Lustre: Skipped 4 previous similar messages [ 671.788720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 671.809700] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 671.831040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 671.837841] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 674.286860] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 679.443112] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 684.507536] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 697.315536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.324473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 697.324685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 697.324689] Lustre: Skipped 2 previous similar messages [ 697.328108] Lustre: Skipped 3 previous similar messages [ 702.453335] Lustre: server umount lustre-OST0000 complete [ 706.027629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.521796] Lustre: server umount lustre-MDT0000 complete [ 712.674718] LustreError: 30966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.696632] LustreError: 30966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 714.476200] LustreError: 29200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436819 with bad export cookie 3575438611846594854 [ 714.484330] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.490944] LustreError: 29200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 714.821843] Lustre: server umount lustre-MDT0001 complete [ 722.174431] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 724.038392] Key type lgssc unregistered [ 724.262644] LNet: 31990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.265896] LNetError: 31990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.277912] LNet: Removed LNI 192.168.204.147@tcp [ 724.876139] Key type .llcrypt unregistered [ 724.877814] Key type ._llcrypt unregistered [ 739.516895] Key type ._llcrypt registered [ 739.518681] Key type .llcrypt registered [ 739.592915] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 740.316935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.433137] alg: No test for adler32 (adler32-zlib) [ 741.414413] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 741.581015] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 743.199410] Key type lgssc registered [ 744.019787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 748.783488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 748.799347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.197235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.934173] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 758.487584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.721684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 761.217449] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 764.740709] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 766.926315] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 771.433894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 771.694718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.418275] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 779.112937] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 782.962295] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 793.524611] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:34:58 (1772436898) [ 794.959757] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 796.245939] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 800.798197] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:35:05 (1772436905) [ 805.726596] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:35:10 (1772436910) [ 806.332374] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 811.135909] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 811.139358] Lustre: Skipped 2 previous similar messages [ 812.085509] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:35:17 (1772436917) [ 816.672828] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 816.680803] Lustre: Skipped 4 previous similar messages [ 825.459479] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 825.464873] Lustre: Skipped 5 previous similar messages [ 827.075925] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:35:31 (1772436931) [ 828.780392] Lustre: *** cfs_fail_loc=131b, val=2*** [ 831.045576] Lustre: *** cfs_fail_loc=131b, val=2*** [ 835.804135] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:35:40 (1772436940) [ 843.933459] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 843.937379] Lustre: Skipped 7 previous similar messages [ 845.167436] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:35:49 (1772436949) [ 855.012775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 855.015957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.040558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.046321] Lustre: Skipped 1 previous similar message [ 860.134373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.137559] Lustre: Skipped 1 previous similar message [ 860.871473] Lustre: server umount lustre-OST0000 complete [ 865.248372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.252893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.265868] Lustre: Skipped 2 previous similar messages [ 869.960563] Lustre: server umount lustre-MDT0000 complete [ 870.375918] LustreError: 33363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.398891] LustreError: 33363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 872.633062] LustreError: 38045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772436977 with bad export cookie 2727365788463276139 [ 872.634059] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.641959] LustreError: 38045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 872.931245] Lustre: server umount lustre-MDT0001 complete [ 879.140361] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 880.713454] Key type lgssc unregistered [ 880.865468] LNet: 40854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.868833] LNetError: 40854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.878245] LNet: Removed LNI 192.168.204.147@tcp [ 881.400961] Key type .llcrypt unregistered [ 881.402234] Key type ._llcrypt unregistered [ 894.757715] Key type ._llcrypt registered [ 894.759702] Key type .llcrypt registered [ 894.857648] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 906.430817] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 906.971749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.043525] alg: No test for adler32 (adler32-zlib) [ 908.029470] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 908.152528] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 909.767292] Key type lgssc registered [ 910.664391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.001382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.017903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.191371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 937.215819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 937.282433] Lustre: lustre-MDT0000: new disk, initializing [ 937.375112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.390230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 940.608857] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 948.285890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.363167] Lustre: 43952:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 948.401685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 948.404431] Lustre: Skipped 1 previous similar message [ 948.451397] Lustre: lustre-MDT0001: new disk, initializing [ 948.503691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 948.534747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 948.542784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 951.319728] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 956.684952] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 959.227988] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 963.557318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.757160] Lustre: lustre-OST0000: new disk, initializing [ 963.760136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 963.860024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.782309] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 973.171277] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.885123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 973.897650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 973.993126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 976.293335] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 976.459325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 979.136893] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 979.344926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 984.551080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 984.552168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.564732] LustreError: Skipped 1 previous similar message [ 984.580899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.597951] Lustre: Skipped 1 previous similar message [ 986.707605] Lustre: server umount lustre-OST0000 complete [ 989.664738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 989.665169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.674394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.681982] Lustre: Skipped 2 previous similar messages [ 989.692550] Lustre: Skipped 1 previous similar message [ 994.789250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.792943] Lustre: Skipped 1 previous similar message [ 995.409421] Lustre: server umount lustre-MDT0000 complete [ 997.978069] LustreError: 43942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437103 with bad export cookie 9925465479303479504 [ 997.980040] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.986399] LustreError: 43942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 998.175775] Lustre: server umount lustre-MDT0001 complete [ 1004.435570] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:38:29 (1772437109) [ 1009.068686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.375985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.505363] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1014.757469] LustreError: 48003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.770432] LustreError: 48003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1018.017532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.250386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.282956] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1025.043795] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1027.646467] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1032.219254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1032.468543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.263831] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1039.735148] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1046.929590] LustreError: 50195:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1047.510038] Lustre: Modifying parameter general.timeout=1 in log params [ 1048.676431] Lustre: Modifying parameter general.timeout=3 in log params [ 1048.679708] Lustre: Skipped 1 previous similar message [ 1051.012078] Lustre: Modifying parameter general.timeout=7 in log params [ 1051.015578] Lustre: Skipped 3 previous similar messages [ 1055.128380] Lustre: Modifying parameter general.timeout=14 in log params [ 1055.130701] Lustre: Skipped 6 previous similar messages [ 1064.091903] Lustre: Modifying parameter general.timeout=28 in log params [ 1064.093861] Lustre: Skipped 13 previous similar messages [ 1080.493922] Lustre: Modifying parameter general.timeout=57 in log params [ 1080.497252] Lustre: Skipped 28 previous similar messages [ 1112.808027] Lustre: Modifying parameter general.timeout=120 in log params [ 1112.809683] Lustre: Skipped 62 previous similar messages [ 1123.143438] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:40:28 (1772437228) [ 1124.796624] Lustre: 57113:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1125.824404] Lustre: 57211:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1128.526723] Lustre: 57522:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1128.532406] Lustre: 57522:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1146.056044] Lustre: 58975:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1146.060793] Lustre: 58975:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1146.070850] LustreError: 58975:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1147.924105] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:40:53 (1772437253) [ 1153.504078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.509067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.511501] Lustre: Skipped 2 previous similar messages [ 1153.524820] Lustre: Skipped 2 previous similar messages [ 1158.631422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.636363] Lustre: Skipped 2 previous similar messages [ 1159.437714] Lustre: server umount lustre-MDT0000 complete [ 1161.232286] LustreError: 52716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437266 with bad export cookie 9925465479303480386 [ 1161.236517] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.236761] LustreError: 52716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1161.380140] Lustre: server umount lustre-MDT0001 complete [ 1173.707743] Lustre: server umount lustre-OST0000 complete [ 1187.016534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.194666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1187.217927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1187.222307] Lustre: Skipped 32 previous similar messages [ 1187.346547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.729044] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1194.784651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.871320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1194.902293] Lustre: 61499:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1197.415465] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1200.139702] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1201.883077] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1205.052986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1205.160168] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1208.024144] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1210.575147] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1211.321315] Lustre: 63723:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1215.456461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.459430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.459711] LustreError: Skipped 1 previous similar message [ 1215.466828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.104279] Lustre: server umount lustre-OST0000 complete [ 1220.576841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1220.577976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.585039] LustreError: 61509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.589509] Lustre: Skipped 2 previous similar messages [ 1222.074759] Lustre: server umount lustre-MDT0000 complete [ 1223.762713] LustreError: 61491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437328 with bad export cookie 9925465479303483032 [ 1223.765144] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.773624] LustreError: 61491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1223.777168] LustreError: 61501:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1223.977406] Lustre: server umount lustre-MDT0001 complete [ 1229.934030] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1231.216299] Key type lgssc unregistered [ 1231.391635] LNet: 64902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.397354] LNetError: 64902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.408556] LNet: Removed LNI 192.168.204.147@tcp [ 1231.800709] Key type .llcrypt unregistered [ 1231.802599] Key type ._llcrypt unregistered [ 1244.308643] Key type ._llcrypt registered [ 1244.309942] Key type .llcrypt registered [ 1244.370438] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1244.854780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1244.907497] alg: No test for adler32 (adler32-zlib) [ 1245.830095] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1245.976138] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1247.591581] Key type lgssc registered [ 1248.212296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1251.931356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1251.940323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.223621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.968935] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1259.136932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.423524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.590383] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1264.394688] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1266.349186] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1269.594362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1269.668106] LustreError: 67875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.675731] LustreError: 67875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1269.851869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.380507] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1274.957248] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1284.535151] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1290.210857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.215230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.221871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.281049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.283424] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.286715] Lustre: Skipped 1 previous similar message [ 1293.292517] Lustre: Skipped 1 previous similar message [ 1294.375165] Lustre: server umount lustre-MDT0000 complete [ 1295.954151] LustreError: 68622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437401 with bad export cookie 10347967581652384857 [ 1295.957771] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1295.958393] LustreError: 68622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1296.076160] Lustre: server umount lustre-MDT0001 complete [ 1306.996368] Lustre: server umount lustre-OST0000 complete [ 1319.343050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.475329] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1319.487051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1319.603575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.317712] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1325.299064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.360703] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1325.377782] Lustre: 70807:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1325.600182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1327.619475] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1330.321456] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1332.105570] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1335.366915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1335.478475] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1335.668980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.479694] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1341.212850] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1342.120373] Lustre: 73042:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1343.144565] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1347.557141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1347.559872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.569527] LustreError: Skipped 1 previous similar message [ 1347.578818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1347.585167] Lustre: Skipped 2 previous similar messages [ 1351.383519] Lustre: server umount lustre-OST0000 complete [ 1356.255850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1356.260797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.266462] Lustre: Skipped 1 previous similar message [ 1356.269661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.529755] Lustre: server umount lustre-MDT0000 complete [ 1361.572102] LustreError: 70799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437466 with bad export cookie 10347967581652385697 [ 1361.572478] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.577892] LustreError: 70799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1361.776356] Lustre: server umount lustre-MDT0001 complete [ 1367.357967] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1368.931879] Key type lgssc unregistered [ 1369.113699] LNet: 74234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.116505] LNetError: 74234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.125752] LNet: Removed LNI 192.168.204.147@tcp [ 1369.603673] Key type .llcrypt unregistered [ 1369.605499] Key type ._llcrypt unregistered [ 1382.796345] Key type ._llcrypt registered [ 1382.797641] Key type .llcrypt registered [ 1382.842559] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1383.309851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1383.329155] alg: No test for adler32 (adler32-zlib) [ 1384.287900] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1384.445736] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1386.048897] Key type lgssc registered [ 1386.618863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1390.904741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1390.922403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.276882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.814851] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1399.828699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.172128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1402.447124] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1405.631502] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1407.449875] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1410.799224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.174853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.748095] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1416.263482] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1429.041918] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1430.089914] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1430.094977] Lustre: Skipped 1 previous similar message [ 1431.283508] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1431.286847] Lustre: Skipped 2 previous similar messages [ 1433.314708] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1433.317909] Lustre: Skipped 4 previous similar messages [ 1437.689423] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1437.693473] Lustre: Skipped 10 previous similar messages [ 1446.012338] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1446.015502] Lustre: Skipped 21 previous similar messages [ 1462.031931] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1462.034769] Lustre: Skipped 43 previous similar messages [ 1470.453580] Lustre: 83358:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1471.548869] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:46:16 (1772437576) [ 1472.214374] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1472.971305] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:46:18 (1772437578) [ 1477.097814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.101489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1477.101857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.105298] Lustre: Skipped 1 previous similar message [ 1480.296484] Lustre: server umount lustre-OST0000 complete [ 1482.210118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.210305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1482.213168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.213174] Lustre: Skipped 1 previous similar message [ 1482.216680] Lustre: Skipped 1 previous similar message [ 1487.328119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.330830] Lustre: Skipped 2 previous similar messages [ 1489.972951] Lustre: server umount lustre-MDT0000 complete [ 1491.405471] LustreError: 75588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437596 with bad export cookie 6643106212286577017 [ 1491.409203] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.412729] LustreError: 75588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1491.551288] Lustre: server umount lustre-MDT0001 complete [ 1495.667988] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1496.902249] Key type lgssc unregistered [ 1497.041984] LNet: 84587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.046472] LNetError: 84587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.055509] LNet: Removed LNI 192.168.204.147@tcp [ 1497.378082] Key type .llcrypt unregistered [ 1497.379380] Key type ._llcrypt unregistered [ 1498.932938] Key type ._llcrypt registered [ 1498.935097] Key type .llcrypt registered [ 1498.979513] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1500.604883] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules [ 1501.057102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1501.098732] alg: No test for adler32 (adler32-zlib) [ 1501.973794] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1502.075448] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1503.671159] Key type lgssc registered [ 1504.190182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1504.632607] LustreError: 85708:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1512.994315] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1517.035984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1525.962103] LustreError: 85708:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1525.981421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.307087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.294305] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1533.585316] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:47:18 (1772437638) [ 1535.849623] LustreError: 86751:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1535.855242] LustreError: 86751:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1535.859576] LustreError: 86751:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 1535.996386] Lustre: server umount lustre-MDT0000 complete [ 1541.555491] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1542.799812] Key type lgssc unregistered [ 1542.946277] LNet: 88207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.951222] LNetError: 88207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.961633] LNet: Removed LNI 192.168.204.147@tcp [ 1543.326052] Key type .llcrypt unregistered [ 1543.327606] Key type ._llcrypt unregistered [ 1551.643421] Key type ._llcrypt registered [ 1551.644812] Key type .llcrypt registered [ 1551.689001] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1552.180563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.202315] alg: No test for adler32 (adler32-zlib) [ 1553.089200] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 1553.207711] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1554.815259] Key type lgssc registered [ 1555.347034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1558.724885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1558.735526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.010624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.763455] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1565.813186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.183298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1568.071483] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1570.510287] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1572.072996] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1575.312648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1575.585921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.935510] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1580.048309] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1616.093798] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:48:41 (1772437721) [ 1616.822770] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1617.536635] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:48:42 (1772437722) [ 1621.986047] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.988782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1621.991822] Lustre: Skipped 2 previous similar messages [ 1621.994501] Lustre: Skipped 1 previous similar message [ 1624.936191] Lustre: server umount lustre-MDT0000 complete [ 1626.611688] LustreError: 89365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437731 with bad export cookie 4588844389583693794 [ 1626.615191] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1626.617499] LustreError: 89365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1626.775132] Lustre: server umount lustre-MDT0001 complete [ 1638.681656] Lustre: server umount lustre-OST0000 complete [ 1643.378302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.584911] LustreError: 93340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1643.594240] LustreError: 93340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1643.654413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.256491] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1648.675437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.967237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1650.504151] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1652.723357] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1654.300952] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1659.287819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1659.376680] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1659.380305] LustreError: 93329:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1659.397072] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1659.401894] LustreError: 95196:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 1659.405313] LustreError: 95196:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1659.408204] LustreError: 95196:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1659.439653] Lustre: server umount lustre-OST0000 complete [ 1659.441221] LustreError: 95196:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1662.733547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1662.817275] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1662.819757] LustreError: 93330:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1662.823220] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1662.825914] LustreError: 95659:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 1662.828637] LustreError: 95659:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1662.830841] LustreError: 95659:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1662.860139] Lustre: server umount lustre-OST0000 complete [ 1662.861924] LustreError: 95659:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1666.294604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1666.362470] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1666.365201] Lustre: Found index 0 for lustre-OST0000, updating log [ 1666.367696] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1666.400954] Lustre: lustre-OST0000: new disk, initializing [ 1666.403838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1666.501847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.561782] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1669.930447] Lustre: Failing over lustre-OST0000 [ 1669.996341] Lustre: server umount lustre-OST0000 complete [ 1673.376536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.381271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.384110] Lustre: Skipped 2 previous similar messages [ 1678.820271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.822586] Lustre: Skipped 1 previous similar message [ 1680.890597] Lustre: server umount lustre-MDT0000 complete [ 1682.353781] LustreError: 93320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437787 with bad export cookie 4588844389583696524 [ 1682.357307] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1682.358149] LustreError: 93320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1682.478473] Lustre: server umount lustre-MDT0001 complete [ 1685.771847] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:49:51 (1772437791) [ 1691.669934] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1695.566099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.726734] LustreError: 98498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.754570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.199559] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1700.373942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.174936] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1703.249848] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1705.780746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.826102] LustreError: 99955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.062095] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1708.142188] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1710.178289] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1710.182368] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:10 [ 1710.189321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1710.194167] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1710.194396] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1710.202641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1711.473749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.509348] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1711.546780] Lustre: lustre-OST0001: new disk, initializing [ 1711.548877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1713.935813] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1716.740469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1716.743915] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1716.788713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1717.412495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1721.379959] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1730.340573] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 1731.552850] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1731.556509] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.562300] Lustre: Skipped 1 previous similar message [ 1735.452536] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 1735.456070] Lustre: Skipped 4 previous similar messages [ 1742.303140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1742.370577] Lustre: server umount lustre-MDT0001 complete [ 1745.692028] LustreError: 98493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1746.495425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.585325] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.587843] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1746.608341] Lustre: 98487:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1746.965885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1746.970254] Lustre: Skipped 3 previous similar messages [ 1748.517694] LustreError: 102446:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1748.520990] LustreError: 102446:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1748.524503] LustreError: 98490:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1748.558122] LustreError: 102454:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.147@tcp: cfg command failed: rc = -17 [ 1748.561922] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1748.561922] [ 1748.797688] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1752.044438] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1752.641640] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 02:50:57 (1772437857) [ 1759.543825] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1767.405271] Lustre: 104333:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1776.860620] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1778.274260] Lustre: 105466:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1833.952129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.952195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1833.953552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.953558] Lustre: Skipped 7 previous similar messages [ 1833.956710] Lustre: Skipped 3 previous similar messages [ 1838.587479] Lustre: server umount lustre-MDT0000 complete [ 1839.072208] LustreError: 98495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.079069] LustreError: 98495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1839.978691] LustreError: 98478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772437945 with bad export cookie 4588844389583697413 [ 1839.980228] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1839.982827] LustreError: 98478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1840.108030] Lustre: server umount lustre-MDT0001 complete [ 1851.397126] Lustre: server umount lustre-OST0000 complete [ 1862.053832] Lustre: server umount lustre-OST0001 complete [ 1864.032122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1867.376838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1870.522283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1874.129392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1881.153808] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1885.147059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.158689] Lustre: lustre-MDT0000: reset Object Index mappings [ 1885.352403] LustreError: 109391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.396680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.797149] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1890.142551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.157195] Lustre: lustre-MDT0001: reset Object Index mappings [ 1892.054776] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1893.236259] Lustre: 110505:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1895.627763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.018328] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1900.007488] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1901.134102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.461566] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1903.618118] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 1910.632466] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1910.632639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 1917.895684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1919.256211] Lustre: 112374:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1941.151493] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:54:06 (1772438046) [ 1972.195212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.195214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1972.195224] Lustre: Skipped 2 previous similar messages [ 1972.197371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.197376] Lustre: Skipped 3 previous similar messages [ 1975.971225] Lustre: server umount lustre-MDT0000 complete [ 1977.218050] LustreError: 109373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438082 with bad export cookie 4588844389583744432 [ 1977.220546] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.221473] LustreError: 109373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1977.313580] LustreError: 112119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.319253] LustreError: 112119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1983.490904] Lustre: server umount lustre-MDT0001 complete [ 1999.327109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1999.384520] Lustre: server umount lustre-OST0000 complete [ 2007.145294] Lustre: server umount lustre-OST0001 complete [ 2009.444510] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2013.025043] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2026.347637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.444550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2026.458796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2026.501254] Lustre: lustre-MDT0000: new disk, initializing [ 2026.529243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.531728] Lustre: Skipped 3 previous similar messages [ 2026.537687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2027.968466] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2031.118621] Lustre: server umount lustre-MDT0000 complete [ 2034.801537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.931433] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2034.936696] Lustre: 116586:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2034.941415] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2036.451950] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2038.367556] Lustre: server umount lustre-MDT0000 complete [ 2041.521730] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:55:46 (1772438146) [ 2042.134526] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2042.763843] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:55:48 (1772438148) [ 2048.785220] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2052.452770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.971943] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2056.972265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.021435] Lustre: 118602:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2057.052884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2057.054822] Lustre: Skipped 1 previous similar message [ 2057.094868] Lustre: lustre-MDT0001: new disk, initializing [ 2057.152899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2057.156820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2058.575103] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2060.889704] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2063.150697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2063.252371] Lustre: lustre-OST0000: new disk, initializing [ 2063.254411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2063.711828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2063.716441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2063.751568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2065.253319] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2069.586643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2070.790301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2070.793957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2070.825146] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2071.703144] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2076.549747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2077.873962] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2223.411648] Lustre: 127354:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2223.414870] Lustre: 127354:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2223.417817] Lustre: 127354:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2223.420096] Lustre: 127354:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2223.422992] Lustre: 127354:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2223.425814] Lustre: 127354:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2318.000417] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:00:23 (1772438423) [ 2332.131056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.134844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.136369] Lustre: Skipped 6 previous similar messages [ 2332.143048] Lustre: Skipped 10 previous similar messages [ 2335.750225] Lustre: server umount lustre-MDT0000 complete [ 2337.132257] LustreError: 118593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438442 with bad export cookie 4588844389583798549 [ 2337.135321] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.136519] LustreError: 118593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2337.248845] LustreError: 118608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.255845] LustreError: 118608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2343.388847] Lustre: server umount lustre-MDT0001 complete [ 2350.789569] Lustre: server umount lustre-OST0000 complete [ 2358.366403] Lustre: server umount lustre-OST0001 complete [ 2360.550377] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2364.000315] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2376.776402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.885503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2376.899496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2376.901751] Lustre: Skipped 1 previous similar message [ 2376.939327] Lustre: lustre-MDT0000: new disk, initializing [ 2376.941387] Lustre: Skipped 1 previous similar message [ 2376.969755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.972275] Lustre: Skipped 5 previous similar messages [ 2376.978185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2378.486129] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2382.795552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.841638] Lustre: 130353:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2382.944127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2384.272432] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2387.508399] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2388.746887] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2391.008513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.187961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2392.228335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2393.070211] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2396.283330] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2399.606914] Lustre: lustre-MDD0000: changelog on [ 2400.598171] Lustre: lustre-MDD0001: changelog on [ 2754.010082] Lustre: 130360:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2781.584966] Lustre: lustre-MDD0001: changelog off [ 2782.635413] Lustre: lustre-MDD0000: changelog off [ 2786.785432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2786.785764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.789234] LustreError: Skipped 1 previous similar message [ 2786.789928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2786.793931] Lustre: Skipped 4 previous similar messages [ 2786.795129] Lustre: Skipped 4 previous similar messages [ 2790.168168] Lustre: server umount lustre-OST0000 complete [ 2797.630628] Lustre: server umount lustre-MDT0000 complete [ 2798.874195] LustreError: 131131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438904 with bad export cookie 4588844389584601778 [ 2798.877912] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.877911] LustreError: 131131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2798.984485] Lustre: server umount lustre-MDT0001 complete [ 2802.245611] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 2803.171907] Key type lgssc unregistered [ 2803.285314] LNet: 140651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2803.287203] LNetError: 140651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2803.297283] LNet: Removed LNI 192.168.204.147@tcp [ 2803.579123] Key type .llcrypt unregistered [ 2803.580612] Key type ._llcrypt unregistered [ 2809.516661] Key type ._llcrypt registered [ 2809.517867] Key type .llcrypt registered [ 2809.551613] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2815.296840] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2815.583477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2815.610023] alg: No test for adler32 (adler32-zlib) [ 2816.454540] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 2816.524712] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 2818.095134] Key type lgssc registered [ 2818.436343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.266741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2829.271575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.364541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2830.376878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2830.413425] Lustre: lustre-MDT0000: new disk, initializing [ 2830.437429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.443783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2831.691901] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2835.755118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.796926] Lustre: 143791:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2835.822550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2835.824203] Lustre: Skipped 1 previous similar message [ 2835.862406] Lustre: lustre-MDT0001: new disk, initializing [ 2835.896860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.908890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2835.912434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2837.168372] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2840.274954] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.498137] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2843.781734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.884773] Lustre: lustre-OST0000: new disk, initializing [ 2843.886564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2843.914795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.846692] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2846.647856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2846.652349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2846.680218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2848.933224] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2851.012895] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_module obdecho/obdecho [ 2851.604263] LustreError: 146405:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2851.606669] LustreError: 146405:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 2852.145361] LustreError: 146501:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2856.415703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2856.417791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.421208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.929667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.933425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.935034] Lustre: Skipped 1 previous similar message [ 2861.210611] Lustre: server umount lustre-MDT0000 complete [ 2862.306303] LustreError: 143782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772438967 with bad export cookie 12180998212164013329 [ 2862.309256] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.309815] LustreError: 143782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2862.406597] Lustre: server umount lustre-MDT0001 complete [ 2873.747109] Lustre: server umount lustre-OST0000 complete [ 2879.757637] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2882.671749] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2893.475091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.565760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2893.578156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2893.616556] Lustre: lustre-MDT0000: new disk, initializing [ 2893.642428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.649661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2895.037811] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2899.220638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.274884] Lustre: 150175:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2899.300726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2899.302616] Lustre: Skipped 1 previous similar message [ 2899.343976] Lustre: lustre-MDT0001: new disk, initializing [ 2899.379919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.394816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2899.398894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2900.696214] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2903.653798] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2904.745639] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2906.781325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.583260] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2908.921498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2908.925538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2908.965099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2911.520720] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2912.749740] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2912.812597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2913.988718] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2914.055780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2919.393580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.395864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2919.396016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2919.397159] Lustre: Skipped 2 previous similar messages [ 2919.399341] LustreError: Skipped 1 previous similar message [ 2920.712675] Lustre: server umount lustre-OST0000 complete [ 2924.512548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.513160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2924.517434] Lustre: Skipped 1 previous similar message [ 2924.520116] Lustre: Skipped 2 previous similar messages [ 2929.631935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2929.634444] Lustre: Skipped 1 previous similar message [ 2936.287101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2936.351633] Lustre: server umount lustre-MDT0000 complete [ 2937.516106] LustreError: 151908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439042 with bad export cookie 12180998212164014274 [ 2937.518527] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.519131] LustreError: 151908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2937.625485] Lustre: server umount lustre-MDT0001 complete [ 2938.746334] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:10:44 (1772439044) [ 2939.240684] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2939.800798] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:10:45 (1772439045) [ 2942.189018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.327813] LustreError: 154174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.334357] LustreError: 154174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2942.349146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.351662] Lustre: Skipped 1 previous similar message [ 2943.522593] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2946.112039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.420048] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2949.073742] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2950.171641] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2952.284151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.031098] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 2955.598719] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2966.305802] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 2966.308361] Lustre: Skipped 2 previous similar messages [ 2967.009510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2967.011789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.018742] Lustre: Skipped 1 previous similar message [ 2967.121238] Lustre: server umount lustre-MDT0000 complete [ 2968.347252] LustreError: 154156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439073 with bad export cookie 12180998212164015086 [ 2968.350621] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2968.351630] LustreError: 154156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2971.419054] LustreError: 154169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.151906] LustreError: 154169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.157317] LustreError: 154169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2974.514397] Lustre: server umount lustre-MDT0001 complete [ 2976.720404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2980.979242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2981.052892] LustreError: 157677:0:(llog.c:1643:llog_backup()) MGC192.168.204.147@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2981.055325] Lustre: 157677:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.147@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2991.642385] LustreError: 157682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.762482] LustreError: 157684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.767103] LustreError: 157684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3008.282237] LustreError: 157699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.286659] LustreError: 157699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3026.911759] LustreError: 157682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.916196] LustreError: 157682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3041.887399] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3041.889461] Lustre: 157677:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3062.554253] LustreError: 157702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.558568] LustreError: 157702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3071.967247] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3071.969461] Lustre: 154166:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.147@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3082.726988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.729333] Lustre: Skipped 2 previous similar messages [ 3084.074768] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3086.737356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.842678] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3088.799112] LustreError: 158400:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3088.801672] LustreError: 158400:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000540ce246 x1858536914115328/t0(0) o250->MGC192.168.204.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1772439193 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3088.807437] LustreError: 158400:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3088.863583] LustreError: 142016:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000082ed1425 x1858536914119552/t0(0) o250->MGC192.168.204.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3090.116856] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3091.701332] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3093.990369] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3094.247646] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3095.370817] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3098.896591] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3104.224269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3104.226232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.226912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.229555] Lustre: Skipped 4 previous similar messages [ 3104.231514] Lustre: Skipped 7 previous similar messages [ 3105.042963] Lustre: server umount lustre-MDT0000 complete [ 3106.176039] LustreError: 154156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439211 with bad export cookie 12180998212164021540 [ 3106.179630] LustreError: 154156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3106.179708] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.282860] Lustre: server umount lustre-MDT0001 complete [ 3117.725729] Lustre: server umount lustre-OST0000 complete [ 3124.543981] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3127.763275] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3139.488473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.584451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3139.595109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3139.597228] Lustre: Skipped 1 previous similar message [ 3139.644225] Lustre: lustre-MDT0000: new disk, initializing [ 3139.646065] Lustre: Skipped 1 previous similar message [ 3139.670887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.672732] Lustre: Skipped 1 previous similar message [ 3139.677695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3141.017068] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3145.069551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.110510] Lustre: 162494:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3145.198607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3146.366846] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3149.203610] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3150.283054] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3152.390030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.190132] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3157.167153] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3158.016470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3158.051382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3158.580415] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3158.663672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3159.941163] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3160.005605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3163.105712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3163.106701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.107437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3163.108787] LustreError: Skipped 1 previous similar message [ 3163.113641] Lustre: Skipped 1 previous similar message [ 3166.745125] Lustre: server umount lustre-OST0000 complete [ 3174.170887] Lustre: server umount lustre-MDT0000 complete [ 3175.394792] LustreError: 165626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439280 with bad export cookie 12180998212164038333 [ 3175.396615] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.400743] LustreError: 165626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3175.523319] Lustre: server umount lustre-MDT0001 complete [ 3178.413547] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:14:43 (1772439283) [ 3180.694286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.843632] LustreError: 166494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.851126] LustreError: 166494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 3182.144046] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3184.994947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.253385] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3187.793899] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3188.855517] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3190.911991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3192.703947] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3194.456411] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3213.966440] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:15:19 (1772439319) [ 3215.840731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.842275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.842633] LustreError: Skipped 1 previous similar message [ 3215.845610] Lustre: Skipped 2 previous similar messages [ 3220.934435] Lustre: server umount lustre-OST0000 complete [ 3231.200527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3231.203081] Lustre: Skipped 10 previous similar messages [ 3236.319113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3236.385944] Lustre: server umount lustre-MDT0000 complete [ 3237.636138] LustreError: 166475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439342 with bad export cookie 12180998212164039208 [ 3237.639489] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.642009] LustreError: 166475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3237.758891] Lustre: server umount lustre-MDT0001 complete [ 3241.230125] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3242.248792] Key type lgssc unregistered [ 3242.380415] LNet: 169886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.382667] LNetError: 169886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.392308] LNet: Removed LNI 192.168.204.147@tcp [ 3242.679205] Key type .llcrypt unregistered [ 3242.680233] Key type ._llcrypt unregistered [ 3250.078033] Key type ._llcrypt registered [ 3250.078994] Key type .llcrypt registered [ 3250.118372] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3250.479735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3250.516548] alg: No test for adler32 (adler32-zlib) [ 3251.369607] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3251.456712] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3253.039106] Key type lgssc registered [ 3253.407370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3256.020269] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3256.027118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3271.455354] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3276.575189] LustreError: 171090:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.147@tcp: failed processing log, type 4: rc = -110 [ 3281.695335] LustreError: 171090:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3281.699046] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3281.700693] LustreError: MGC192.168.204.147@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3281.703757] LustreError: 171090:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3281.706335] LustreError: 171090:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 3281.708450] LustreError: 171090:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 3281.730896] Lustre: server umount lustre-OST0000 complete [ 3281.732269] LustreError: 171090:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3284.623289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.780522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.143075] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3290.092267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.215213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3291.527829] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3293.243155] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3294.423704] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3296.618315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.697144] LustreError: 173177:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3296.703899] Lustre: 173177:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.204.147@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3296.755049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.637486] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3300.386359] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3305.439702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3305.441982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.445607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.953287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.954163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.957371] Lustre: Skipped 1 previous similar message [ 3305.959703] Lustre: Skipped 1 previous similar message [ 3307.152167] Lustre: server umount lustre-MDT0000 complete [ 3308.260141] LustreError: 171614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439413 with bad export cookie 6706807692387619800 [ 3308.263979] LustreError: 171614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3308.264185] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.372743] Lustre: server umount lustre-MDT0001 complete [ 3318.932743] Lustre: server umount lustre-OST0000 complete [ 3321.790802] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3325.016353] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3337.179207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.273389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3337.287904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3337.332862] Lustre: lustre-MDT0000: new disk, initializing [ 3337.360796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.368051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3338.712818] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3342.911256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.962060] Lustre: 176550:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3342.989608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3342.991334] Lustre: Skipped 1 previous similar message [ 3343.031991] Lustre: lustre-MDT0001: new disk, initializing [ 3343.073443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3343.087341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3343.091242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3344.449355] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3347.528938] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3348.596220] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3350.727451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.821234] Lustre: lustre-OST0000: new disk, initializing [ 3350.823101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3352.499246] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3355.336579] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3356.502955] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3356.660473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3356.663319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3356.694855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3357.596313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3358.916715] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3358.983337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3361.762122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.762391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.764310] LustreError: Skipped 1 previous similar message [ 3361.764789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3361.768014] Lustre: Skipped 1 previous similar message [ 3361.772593] Lustre: Skipped 1 previous similar message [ 3365.715136] Lustre: server umount lustre-OST0000 complete [ 3368.927850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.930096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.934119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.120061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.121830] Lustre: Skipped 3 previous similar messages [ 3381.215193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3381.274704] Lustre: server umount lustre-MDT0000 complete [ 3382.240251] LustreError: 176560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.245435] LustreError: 176560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3382.530058] LustreError: 176540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439487 with bad export cookie 6706807692387620486 [ 3382.533776] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3382.534676] LustreError: 176540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3382.651524] Lustre: server umount lustre-MDT0001 complete [ 3385.512928] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:18:10 (1772439490) [ 3391.070693] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3391.997488] Key type lgssc unregistered [ 3392.118342] LNet: 181073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3392.120430] LNetError: 181073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3392.129261] LNet: Removed LNI 192.168.204.147@tcp [ 3392.411074] Key type .llcrypt unregistered [ 3392.411974] Key type ._llcrypt unregistered [ 3398.990346] Key type ._llcrypt registered [ 3398.991387] Key type .llcrypt registered [ 3399.036089] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3399.442926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3399.457401] alg: No test for adler32 (adler32-zlib) [ 3400.308282] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3400.389278] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3401.967165] Key type lgssc registered [ 3402.394906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3405.163508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3405.169294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.309031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.495964] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3410.189633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.304602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3411.516260] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3413.180463] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3414.224466] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3416.597704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3416.661844] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3416.662853] LustreError: 183883:0:(llog.c:1684:llog_backup()) MGC192.168.204.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3416.664976] Lustre: 183883:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.204.147@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3416.695924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.337176] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3419.933259] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3420.639644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3420.641356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.644270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.664448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.665752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.671851] Lustre: Skipped 1 previous similar message [ 3426.709377] Lustre: server umount lustre-MDT0000 complete [ 3426.785065] LustreError: 182250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.790214] LustreError: 182250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3427.840684] LustreError: 182233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439532 with bad export cookie 9411732472504382846 [ 3427.844342] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.846398] LustreError: 182233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.855217] LustreError: 182961:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000343d9ecb x1858537526412928/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3429.901440] Lustre: server umount lustre-MDT0001 complete [ 3441.295113] Lustre: server umount lustre-OST0000 complete [ 3444.318423] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3447.162962] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3457.642225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3457.724598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3457.734705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3457.766195] Lustre: lustre-MDT0000: new disk, initializing [ 3457.787097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.793733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3458.938082] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3462.942276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3462.981552] Lustre: 187266:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3462.999670] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3463.001194] Lustre: Skipped 1 previous similar message [ 3463.002832] LustreError: 188030:0:(llog.c:1684:llog_backup()) MGC192.168.204.147@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3463.006425] LustreError: 188030:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3463.009307] Lustre: 188030:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.147@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3463.017846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3463.020449] Lustre: Skipped 1 previous similar message [ 3463.055409] Lustre: lustre-MDT0001: new disk, initializing [ 3463.087768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3463.098463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3463.102590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3464.425702] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3467.534507] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3468.710965] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3470.981265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.077274] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3471.078721] Lustre: Skipped 1 previous similar message [ 3471.079992] LustreError: 188983:0:(llog.c:1684:llog_backup()) MGC192.168.204.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3471.082915] LustreError: 188983:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3471.084889] Lustre: 188983:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.147@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3471.089150] Lustre: 188983:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3471.097434] Lustre: lustre-OST0000: new disk, initializing [ 3471.099084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3471.124580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.060094] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3475.948063] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3476.978891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3476.981579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3477.010866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3477.146494] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3477.204213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3478.469378] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3478.540620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3482.081930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3482.083731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.084853] LustreError: Skipped 1 previous similar message [ 3482.089295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3482.092273] Lustre: Skipped 3 previous similar messages [ 3485.262220] Lustre: server umount lustre-OST0000 complete [ 3487.201357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.202750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3487.205121] Lustre: Skipped 2 previous similar messages [ 3492.320129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3492.322441] Lustre: Skipped 2 previous similar messages [ 3492.763315] Lustre: server umount lustre-MDT0000 complete [ 3494.100263] LustreError: 188044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439599 with bad export cookie 9411732472504383546 [ 3494.103741] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3494.104207] LustreError: 188044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3494.213587] Lustre: server umount lustre-MDT0001 complete [ 3496.863579] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:20:02 (1772439602) [ 3501.811950] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3504.996546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.122291] LustreError: 191963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.137136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.376794] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3509.076398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.401370] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3511.321856] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3513.387399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.237194] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3517.884577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.944874] Lustre: lustre-OST0001: new disk, initializing [ 3517.946717] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3519.885807] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3525.016879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3526.144972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3526.149440] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3526.176538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3531.615342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3537.270969] Lustre: DEBUG MARKER: ADD OST3 [ 3539.031888] loop: module loaded [ 3539.055887] loop0: detected capacity change from 0 to 204800000 [ 3539.128948] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3539.134781] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3539.139174] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3541.249453] loop0: detected capacity change from 0 to 204800000 [ 3543.290950] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3543.366838] Lustre: lustre-OST0003: new disk, initializing [ 3543.369358] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3543.419668] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3543.423400] Lustre: Skipped 3 previous similar messages [ 3544.162053] Lustre: DEBUG MARKER: STOP OST3 [ 3544.881464] Lustre: Failing over lustre-OST0003 [ 3544.927978] Lustre: server umount lustre-OST0003 complete [ 3545.036797] LustreError: 193414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.689816] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3548.709357] LustreError: 193412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.715251] LustreError: 193412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3549.626802] Lustre: DEBUG MARKER: START OST3 again [ 3551.998115] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3552.098221] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3553.703964] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3553.712483] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3553.839891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3553.843151] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3553.856108] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3554.049195] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3567.389411] Lustre: Failing over lustre-OST0003 [ 3567.423432] Lustre: server umount lustre-OST0003 complete [ 3567.587736] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.592537] LustreError: 193412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.596378] LustreError: 193412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3577.828842] LustreError: 193414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.832595] LustreError: 193414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3580.895977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3580.899355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.903661] Lustre: Skipped 1 previous similar message [ 3580.905498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3585.955584] Lustre: server umount lustre-MDT0000 complete [ 3587.268484] LustreError: 193063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439692 with bad export cookie 9411732472504384414 [ 3587.274087] LustreError: 193063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3587.274447] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3593.532392] Lustre: server umount lustre-MDT0001 complete [ 3594.987666] Lustre: server umount lustre-OST0000 complete [ 3596.353707] Lustre: server umount lustre-OST0001 complete [ 3598.239694] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3600.506965] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3613.220746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.317558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3613.329335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3613.371516] Lustre: lustre-MDT0000: new disk, initializing [ 3613.397232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.399594] Lustre: Skipped 1 previous similar message [ 3613.405920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3614.773745] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3618.843033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.880920] Lustre: 201284:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3618.970629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3620.184659] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3623.070551] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3624.144345] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3626.372170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.215627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3628.253175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3628.379215] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3631.274299] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3632.597896] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3632.666937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3633.915532] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3633.996229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3638.756179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3638.756425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.758524] LustreError: Skipped 1 previous similar message [ 3638.759090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3638.762126] Lustre: Skipped 6 previous similar messages [ 3638.766427] Lustre: Skipped 9 previous similar messages [ 3640.713430] Lustre: server umount lustre-OST0000 complete [ 3656.159102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3656.217193] Lustre: server umount lustre-MDT0000 complete [ 3657.343172] LustreError: 201276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439762 with bad export cookie 9411732472504386675 [ 3657.343657] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3657.346827] LustreError: 201276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3657.459243] Lustre: server umount lustre-MDT0001 complete [ 3658.617761] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:22:44 (1772439764) [ 3663.604325] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3666.389705] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3677.478934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.563461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3677.573547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3677.575463] Lustre: Skipped 3 previous similar messages [ 3677.611367] Lustre: lustre-MDT0000: new disk, initializing [ 3677.613078] Lustre: Skipped 2 previous similar messages [ 3677.640797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3677.644338] Lustre: Skipped 2 previous similar messages [ 3678.853717] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3682.755364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.793502] Lustre: 207443:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3682.884742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3684.064370] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3686.981929] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3688.034897] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3690.083088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.859549] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3691.928047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3694.692226] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3695.906137] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3695.968264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3697.161624] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3697.226926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3702.240747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3702.241592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.243670] LustreError: Skipped 1 previous similar message [ 3702.248386] Lustre: Skipped 2 previous similar messages [ 3702.249115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3702.253229] Lustre: Skipped 7 previous similar messages [ 3703.957055] Lustre: server umount lustre-OST0000 complete [ 3711.467188] Lustre: server umount lustre-MDT0000 complete [ 3712.481599] LustreError: 207455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.488957] LustreError: 207455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3712.789358] LustreError: 207434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439817 with bad export cookie 9411732472504387564 [ 3712.793203] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3712.794191] LustreError: 207434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3712.909439] Lustre: server umount lustre-MDT0001 complete [ 3715.763090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.284804] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3720.176571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.809234] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3723.621804] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3724.831760] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3727.099056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.075030] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3730.925339] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3750.879965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3753.877537] Lustre: server umount lustre-OST0000 complete [ 3761.304346] Lustre: server umount lustre-MDT0000 complete [ 3762.144110] LustreError: 211266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.148739] LustreError: 211266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3762.521347] LustreError: 211229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772439867 with bad export cookie 9411732472504388446 [ 3762.524819] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3762.526175] LustreError: 211229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3762.631931] Lustre: server umount lustre-MDT0001 complete [ 3863.491643] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3864.533564] Key type lgssc unregistered [ 3864.666325] LNet: 214577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3864.668780] LNetError: 214577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3864.679304] LNet: Removed LNI 192.168.204.147@tcp [ 3864.979150] Key type .llcrypt unregistered [ 3864.980182] Key type ._llcrypt unregistered [ 3873.144482] Key type ._llcrypt registered [ 3873.145948] Key type .llcrypt registered [ 3873.184699] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3878.798802] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3879.128588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3879.157597] alg: No test for adler32 (adler32-zlib) [ 3879.999823] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 3880.069164] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3881.647093] Key type lgssc registered [ 3881.949272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3892.689030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3892.697880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.786354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3893.798643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3893.831369] Lustre: lustre-MDT0000: new disk, initializing [ 3893.853216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3893.876254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3895.084370] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3899.030677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.070139] Lustre: 217870:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3899.091479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3899.093165] Lustre: Skipped 1 previous similar message [ 3899.126585] Lustre: lustre-MDT0001: new disk, initializing [ 3899.156460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3899.168697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3899.172270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3900.372048] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3903.378194] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3904.539672] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3906.752084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3906.850980] Lustre: lustre-OST0000: new disk, initializing [ 3906.853545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3906.883977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.650219] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3910.645406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3910.648536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3910.676351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3911.526490] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3912.706926] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3912.769752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3913.955136] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3914.018268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3915.747245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.747497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3915.747916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.750353] Lustre: Skipped 1 previous similar message [ 3915.756716] LustreError: Skipped 1 previous similar message [ 3916.690068] Lustre: server umount lustre-OST0000 complete [ 3919.839710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3919.841595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.845106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.846783] Lustre: Skipped 1 previous similar message [ 3920.864121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3920.865066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.984108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.985770] Lustre: Skipped 2 previous similar messages [ 3931.104045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.105782] Lustre: Skipped 1 previous similar message [ 3932.127116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3932.196142] Lustre: server umount lustre-MDT0000 complete [ 3933.449640] LustreError: 218646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440038 with bad export cookie 8630335620223330726 [ 3933.453860] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3933.584782] Lustre: server umount lustre-MDT0001 complete [ 3939.364733] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3942.922587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.093251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.451506] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3947.306261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.436175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3948.756861] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3949.719674] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3951.914688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3953.885743] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3956.668476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.733437] Lustre: lustre-OST0001: new disk, initializing [ 3956.735959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3956.768704] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.770559] Lustre: Skipped 1 previous similar message [ 3958.735074] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 3960.323051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3960.325957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3960.362988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3964.093975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3971.844905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3980.649952] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:28:06 (1772440086) [ 3983.328781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3983.332230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.337724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3987.482603] Lustre: server umount lustre-MDT0000 complete [ 3988.735273] LustreError: 222354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440093 with bad export cookie 8630335620223331594 [ 3988.739154] LustreError: 222354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3988.739477] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.878110] Lustre: server umount lustre-MDT0001 complete [ 4000.259264] Lustre: server umount lustre-OST0000 complete [ 4011.739830] Lustre: server umount lustre-OST0001 complete [ 4013.764436] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4016.824964] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4028.380354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.486029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4028.496966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4028.534732] Lustre: lustre-MDT0000: new disk, initializing [ 4028.561605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.567882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4029.875893] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4033.887873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.931386] Lustre: 228516:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4034.032784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4035.336658] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4038.347222] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4039.526819] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4041.671258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.769710] Lustre: lustre-OST0000: new disk, initializing [ 4041.772050] Lustre: Skipped 1 previous similar message [ 4041.773780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4041.776561] Lustre: Skipped 2 previous similar messages [ 4043.450630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4043.452957] Lustre: Skipped 1 previous similar message [ 4043.454774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4043.489069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4043.634478] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4046.668592] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4048.088759] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4048.168534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4049.553435] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4049.626447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4053.986223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.986578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4053.992143] Lustre: Skipped 3 previous similar messages [ 4053.994630] LustreError: Skipped 1 previous similar message [ 4053.995332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4053.999543] Lustre: Skipped 5 previous similar messages [ 4056.336215] Lustre: server umount lustre-OST0000 complete [ 4063.703844] Lustre: server umount lustre-MDT0000 complete [ 4064.225030] LustreError: 228527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.231514] LustreError: 228527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4064.983134] LustreError: 228506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440170 with bad export cookie 8630335620223333694 [ 4064.987366] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4064.988806] LustreError: 228506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4065.113317] Lustre: server umount lustre-MDT0001 complete [ 4067.998298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.162877] LustreError: 232320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.186357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.188343] Lustre: Skipped 2 previous similar messages [ 4069.396644] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4072.174370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.634279] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4075.452069] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4076.649730] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4078.876195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.846245] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4082.621475] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4226.529492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4226.530443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4226.535356] Lustre: Skipped 2 previous similar messages [ 4226.538321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4226.540873] Lustre: Skipped 3 previous similar messages [ 4231.755860] Lustre: server umount lustre-OST0000 complete [ 4236.255835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4239.131485] Lustre: server umount lustre-MDT0000 complete [ 4240.385162] LustreError: 232300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440345 with bad export cookie 8630335620223334576 [ 4240.388156] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4240.390380] LustreError: 232300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4240.502823] Lustre: server umount lustre-MDT0001 complete [ 4347.488647] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4348.460528] Key type lgssc unregistered [ 4348.579400] LNet: 235574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4348.582209] LNetError: 235574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4348.589305] LNet: Removed LNI 192.168.204.147@tcp [ 4348.873114] Key type .llcrypt unregistered [ 4348.874973] Key type ._llcrypt unregistered [ 4356.867613] Key type ._llcrypt registered [ 4356.868657] Key type .llcrypt registered [ 4356.913633] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4362.967602] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4363.467227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4363.489728] alg: No test for adler32 (adler32-zlib) [ 4364.338622] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 4364.428767] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4366.031195] Key type lgssc registered [ 4366.442225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4380.328487] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4383.553879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4383.561215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.646828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4384.657230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4384.693066] Lustre: lustre-MDT0000: new disk, initializing [ 4384.717088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.723549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4385.934241] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4389.949707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.000803] Lustre: 239564:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4390.029897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4390.031664] Lustre: Skipped 1 previous similar message [ 4390.068768] Lustre: lustre-MDT0001: new disk, initializing [ 4390.101136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4390.114441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4390.117613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4391.320413] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4393.565627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4395.716284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4395.815285] Lustre: lustre-OST0000: new disk, initializing [ 4395.818079] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4395.848125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.176692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4397.179623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4397.224966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4397.640532] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4401.577732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4401.640180] Lustre: lustre-OST0001: new disk, initializing [ 4401.641908] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4401.673923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4403.453031] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4404.613833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4404.616838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4404.654050] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4408.711692] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4412.282310] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4421.087936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4421.090138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4421.093783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4425.186119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.188785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4425.191435] Lustre: Skipped 2 previous similar messages [ 4425.193034] Lustre: Skipped 2 previous similar messages [ 4426.589248] Lustre: server umount lustre-MDT0000 complete [ 4427.781163] LustreError: 239555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440532 with bad export cookie 11318533200491049007 [ 4427.784850] LustreError: 239555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4427.784875] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4427.892857] Lustre: server umount lustre-MDT0001 complete [ 4439.564247] Lustre: server umount lustre-OST0000 complete [ 4450.024941] Lustre: server umount lustre-OST0001 complete [ 4454.498773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4457.605321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4460.405877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4463.413217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4469.343410] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4472.631534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.640057] Lustre: lustre-MDT0000: reset Object Index mappings [ 4472.816422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.068928] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4476.776024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.783437] Lustre: lustre-MDT0001: reset Object Index mappings [ 4478.309432] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4479.248032] Lustre: 248091:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4481.344670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4481.461326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.463054] Lustre: Skipped 1 previous similar message [ 4482.469732] LustreError: 248445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.474800] LustreError: 248445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4482.475811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4483.326341] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4486.067777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.209917] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4487.986674] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4491.122976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4492.378837] Lustre: 249932:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4502.501094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.505429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.511329] Lustre: Skipped 2 previous similar messages [ 4502.517509] Lustre: Skipped 2 previous similar messages [ 4502.522036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4504.666195] Lustre: server umount lustre-MDT0000 complete [ 4505.918178] LustreError: 247708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440611 with bad export cookie 11318533200491055783 [ 4505.921536] LustreError: 247708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4505.922225] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4506.037566] Lustre: server umount lustre-MDT0001 complete [ 4517.829840] Lustre: server umount lustre-OST0000 complete [ 4529.312183] Lustre: server umount lustre-OST0001 complete [ 4535.277704] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:37:20 (1772440640) [ 4540.370581] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4543.496856] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4555.415586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.530230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4555.541153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4555.577474] Lustre: lustre-MDT0000: new disk, initializing [ 4555.602541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.604355] Lustre: Skipped 1 previous similar message [ 4555.609961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4556.989209] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4561.531065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.580444] Lustre: 254148:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4561.712685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4563.226067] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4566.334560] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4567.491724] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4569.666612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4569.774422] Lustre: lustre-OST0000: new disk, initializing [ 4569.775765] Lustre: Skipped 1 previous similar message [ 4569.777528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4569.779047] Lustre: Skipped 2 previous similar messages [ 4570.935468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4570.938595] Lustre: Skipped 1 previous similar message [ 4570.940072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4570.980647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4571.559149] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4574.530282] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4575.809208] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4575.879708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4577.027280] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4577.087075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4581.345983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.346439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.347859] LustreError: Skipped 1 previous similar message [ 4581.348425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4581.350997] Lustre: Skipped 2 previous similar messages [ 4581.352033] Lustre: Skipped 1 previous similar message [ 4583.762824] Lustre: server umount lustre-OST0000 complete [ 4586.464843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.465584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.469875] Lustre: Skipped 1 previous similar message [ 4596.704085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4596.706250] Lustre: Skipped 4 previous similar messages [ 4599.263083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.320686] Lustre: server umount lustre-MDT0000 complete [ 4600.463544] LustreError: 254138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440705 with bad export cookie 11318533200491057876 [ 4600.466444] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.468783] LustreError: 254138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4600.584229] Lustre: server umount lustre-MDT0001 complete [ 4605.761347] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4608.861195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.987599] LustreError: 258652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.003386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.005097] Lustre: Skipped 2 previous similar messages [ 4610.203976] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4613.012521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.364159] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4615.302565] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4617.474482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4619.414899] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4622.160637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4622.234660] Lustre: lustre-OST0001: new disk, initializing [ 4622.236522] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4623.308951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4623.312608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4623.358067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4624.142126] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4628.403757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4629.485962] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4636.257545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4636.262156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4638.690945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.691735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4638.692344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.696399] Lustre: Skipped 3 previous similar messages [ 4644.380385] Lustre: server umount lustre-MDT0000 complete [ 4645.522323] LustreError: 258633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440750 with bad export cookie 11318533200491058758 [ 4645.525379] LustreError: 258633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4645.525883] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4645.640166] Lustre: server umount lustre-MDT0001 complete [ 4656.132602] Lustre: server umount lustre-OST0000 complete [ 4667.616768] Lustre: server umount lustre-OST0001 complete [ 4672.871194] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4676.486522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.647518] LustreError: 263926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.669985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.671969] Lustre: Skipped 3 previous similar messages [ 4677.949650] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4680.705478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.126805] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4683.092996] Lustre: 265033:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4685.172919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.308541] LustreError: 265386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.314155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4687.097126] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4689.801915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4690.932596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4691.786716] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4694.814716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4695.960434] Lustre: 266880:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4702.540318] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4702.543600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4705.510035] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:40:10 (1772440810) [ 4717.024051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4717.027775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4717.034376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4717.037272] Lustre: Skipped 7 previous similar messages [ 4723.110661] Lustre: server umount lustre-MDT0000 complete [ 4724.383484] LustreError: 266882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440829 with bad export cookie 11318533200491065065 [ 4724.387258] LustreError: 266882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4724.389369] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4726.752198] LustreError: 263923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4726.757316] LustreError: 263923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4730.537347] Lustre: server umount lustre-MDT0001 complete [ 4746.207078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4746.252605] Lustre: server umount lustre-OST0000 complete [ 4753.492613] Lustre: server umount lustre-OST0001 complete [ 4755.460068] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4758.515337] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4773.028029] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4776.425192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.517246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4776.527688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4776.564581] Lustre: lustre-MDT0000: new disk, initializing [ 4776.593293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4777.845968] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4781.833636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.887196] Lustre: 271063:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4781.992055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4783.267092] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4787.636574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4787.920277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4789.622087] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4793.575058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4795.448047] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4795.466310] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4799.794785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4810.267120] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 03:41:55 (1772440915) [ 4812.767919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4812.771079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.777817] Lustre: Skipped 5 previous similar messages [ 4812.780271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4812.782714] Lustre: Skipped 8 previous similar messages [ 4817.124778] Lustre: server umount lustre-MDT0000 complete [ 4818.547342] LustreError: 271053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772440923 with bad export cookie 11318533200491072954 [ 4818.550762] LustreError: 271053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4818.551247] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4818.667910] Lustre: server umount lustre-MDT0001 complete [ 4830.212147] Lustre: server umount lustre-OST0000 complete [ 4841.701780] Lustre: server umount lustre-OST0001 complete [ 4847.689327] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4850.510434] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4862.415427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.499140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4862.501925] Lustre: Skipped 2 previous similar messages [ 4862.509767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4862.511291] Lustre: Skipped 4 previous similar messages [ 4862.545839] Lustre: lustre-MDT0000: new disk, initializing [ 4862.547718] Lustre: Skipped 3 previous similar messages [ 4862.571597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.575085] Lustre: Skipped 7 previous similar messages [ 4862.581350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4862.583451] Lustre: Skipped 3 previous similar messages [ 4863.794162] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4867.687227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.727524] Lustre: 278092:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4867.825901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4867.829226] Lustre: Skipped 2 previous similar messages [ 4869.019860] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4872.102814] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4873.200226] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4875.366742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4876.568802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4877.403722] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 4880.525569] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4881.204541] Lustre: Mounted lustre-client [ 4881.557005] LustreError: 280589:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffffa034c910a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4881.590194] Lustre: Unmounted lustre-client [ 4886.499906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.507319] Lustre: Skipped 4 previous similar messages [ 4886.604442] Lustre: server umount lustre-OST0000 complete [ 4888.543789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4902.367138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4902.441780] Lustre: server umount lustre-MDT0000 complete [ 4903.637100] LustreError: 278082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441008 with bad export cookie 11318533200491074480 [ 4903.639732] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.640033] LustreError: 278082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4903.753119] Lustre: server umount lustre-MDT0001 complete [ 4907.271924] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4908.214981] Key type lgssc unregistered [ 4943.007147] LNet: 1 local NIs in recovery (showing 1): 192.168.204.147@tcp [ 4943.135330] LNet: 281866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4943.137385] LNetError: 281866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5010.725332] LNet: Removed LNI 192.168.204.147@tcp [ 5011.002103] Key type .llcrypt unregistered [ 5011.003253] Key type ._llcrypt unregistered [ 5016.166726] Key type ._llcrypt registered [ 5016.168037] Key type .llcrypt registered [ 5016.199825] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 5023.043279] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5023.420907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5023.426325] alg: No test for adler32 (adler32-zlib) [ 5024.285337] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5024.380564] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5025.967103] Key type lgssc registered [ 5026.357373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5036.423430] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:45:41 (1772441141) [ 5038.475151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5038.479842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5039.589064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5039.599989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5039.635797] Lustre: lustre-MDT0000: new disk, initializing [ 5039.658604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5039.664599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5040.861714] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5044.746977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5044.789684] Lustre: 285027:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5044.814465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5044.816024] Lustre: Skipped 1 previous similar message [ 5044.854292] Lustre: lustre-MDT0001: new disk, initializing [ 5044.887170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5044.898488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5044.901708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5046.094466] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5048.999257] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5050.052136] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5052.200563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5052.313066] Lustre: lustre-OST0000: new disk, initializing [ 5052.315213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5052.347085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.978840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5052.981551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5053.016905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5054.312547] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5057.501284] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5063.138260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5063.138455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.140796] LustreError: Skipped 1 previous similar message [ 5063.141384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5063.145778] Lustre: Skipped 1 previous similar message [ 5063.152148] Lustre: Skipped 1 previous similar message [ 5067.407141] Lustre: server umount lustre-OST0000 complete [ 5070.815799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5070.817787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5070.821215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5073.376171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5073.376593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.645601] Lustre: server umount lustre-MDT0000 complete [ 5075.737243] LustreError: 285805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441180 with bad export cookie 11174759733348853227 [ 5075.739689] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.741438] LustreError: 285805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5075.855654] Lustre: server umount lustre-MDT0001 complete [ 5079.156654] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5080.144099] Key type lgssc unregistered [ 5080.260301] LNet: 288396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5080.262428] LNetError: 288396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5080.270279] LNet: Removed LNI 192.168.204.147@tcp [ 5080.647192] Key type .llcrypt unregistered [ 5080.648975] Key type ._llcrypt unregistered [ 5089.613491] Key type ._llcrypt registered [ 5089.614375] Key type .llcrypt registered [ 5089.644940] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5089.929460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5089.970204] alg: No test for adler32 (adler32-zlib) [ 5090.807113] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5090.879223] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5092.455125] Key type lgssc registered [ 5092.777269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5095.186134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5095.192637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.356343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.630582] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5100.862126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.008981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5102.721454] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5105.153308] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5106.644786] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5109.381876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5109.517720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5111.964909] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5113.851253] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.904749] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5127.387439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5129.509703] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 5130.834730] Lustre: server umount lustre-MDT0000 complete [ 5131.743783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5131.746337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.750581] LustreError: 289772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.756042] LustreError: 289772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5131.993171] LustreError: 291387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441237 with bad export cookie 8120651866917397895 [ 5131.996976] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.256182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.256818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5132.259307] Lustre: Skipped 1 previous similar message [ 5132.259697] LustreError: 292044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.115670] Lustre: server umount lustre-MDT0001 complete [ 5139.743697] Lustre: lustre-OST0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 5141.382680] Lustre: server umount lustre-OST0000 complete [ 5144.016325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.285675] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5146.207669] Lustre: 293519:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.147@tcp [ 5146.836100] Lustre: server umount MGS complete [ 5149.245052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.383319] LustreError: 294089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.402713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5150.499665] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5152.932251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.143786] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5156.548390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.648798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.650543] Lustre: Skipped 1 previous similar message [ 5158.231462] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5159.745497] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5160.196821] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5161.173368] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5165.760727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5169.121492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5169.121657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5169.125392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5169.130080] Lustre: Skipped 1 previous similar message [ 5172.872191] Lustre: server umount lustre-OST0000 complete [ 5174.240546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5174.241432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5174.244027] Lustre: Skipped 2 previous similar messages [ 5174.248044] Lustre: Skipped 1 previous similar message [ 5180.113158] Lustre: server umount lustre-MDT0000 complete [ 5181.193125] LustreError: 294808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441286 with bad export cookie 8120651866917398721 [ 5181.195178] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.195988] LustreError: 294808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5181.301770] Lustre: server umount lustre-MDT0001 complete [ 5184.467291] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5185.481884] Key type lgssc unregistered [ 5185.614381] LNet: 297268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5185.616389] LNetError: 297268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5185.625441] LNet: Removed LNI 192.168.204.147@tcp [ 5185.933040] Key type .llcrypt unregistered [ 5185.934102] Key type ._llcrypt unregistered [ 5191.480978] Key type ._llcrypt registered [ 5191.482009] Key type .llcrypt registered [ 5191.516391] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 5197.192802] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5197.577685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5197.584851] alg: No test for adler32 (adler32-zlib) [ 5198.434783] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5198.515077] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5200.095108] Key type lgssc registered [ 5200.440750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5210.568599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5210.574857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5211.666549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5211.676694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5211.716255] Lustre: lustre-MDT0000: new disk, initializing [ 5211.740205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.745974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5212.952626] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5216.793199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.829635] Lustre: 300364:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5216.854780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5216.856496] Lustre: Skipped 1 previous similar message [ 5216.894633] Lustre: lustre-MDT0001: new disk, initializing [ 5216.928439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5216.939355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5216.942152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5218.115792] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5220.913484] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5221.939805] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5223.993750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5224.092445] Lustre: lustre-OST0000: new disk, initializing [ 5224.094115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5224.125557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.022298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5225.025425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5225.076875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5225.836114] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all [ 5228.692452] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5229.854721] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5229.911930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5231.243890] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5231.304295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5235.168207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5235.171244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.175986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5235.680567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5235.682496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.685958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5237.965601] Lustre: server umount lustre-OST0000 complete [ 5240.800821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.801765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5240.805039] Lustre: Skipped 1 previous similar message [ 5240.811180] Lustre: Skipped 1 previous similar message [ 5245.272055] Lustre: server umount lustre-MDT0000 complete [ 5246.365230] LustreError: 300354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772441351 with bad export cookie 10872684491722785844 [ 5246.368114] LustreError: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5246.370355] LustreError: 300354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5246.471663] Lustre: server umount lustre-MDT0001 complete [ 5249.019787] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 03:49:14 (1772441354) [ 5264.560533] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 03:49:29 (1772441369) [ 5277.442037] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5278.282815] Key type lgssc unregistered [ 5278.397342] LNet: 306898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5278.399468] LNetError: 306898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5278.410312] LNet: Removed LNI 192.168.204.147@tcp [ 5278.731124] Key type .llcrypt unregistered [ 5278.732148] Key type ._llcrypt unregistered [ 5304.849631] Key type ._llcrypt registered [ 5304.850588] Key type .llcrypt registered [ 5304.882099] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5304.968206] Key type .llcrypt unregistered [ 5304.969767] Key type ._llcrypt unregistered [ 5314.936698] Key type ._llcrypt registered [ 5314.937650] Key type .llcrypt registered [ 5314.969216] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5315.051184] Key type .llcrypt unregistered [ 5315.052069] Key type ._llcrypt unregistered [ 5326.186378] Key type ._llcrypt registered [ 5326.187417] Key type .llcrypt registered [ 5326.218642] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5326.300207] Key type .llcrypt unregistered [ 5326.301165] Key type ._llcrypt unregistered [ 5352.045542] Key type ._llcrypt registered [ 5352.046844] Key type .llcrypt registered [ 5352.079456] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5352.164310] Key type .llcrypt unregistered [ 5352.165172] Key type ._llcrypt unregistered [ 5361.965329] Key type ._llcrypt registered [ 5361.966310] Key type .llcrypt registered [ 5362.005708] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5362.103353] Key type .llcrypt unregistered [ 5362.104304] Key type ._llcrypt unregistered [ 5371.732313] Key type ._llcrypt registered [ 5371.733265] Key type .llcrypt registered [ 5371.773174] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5371.871345] Key type .llcrypt unregistered [ 5371.872343] Key type ._llcrypt unregistered [ 5381.578466] Key type ._llcrypt registered [ 5381.579440] Key type .llcrypt registered [ 5381.616782] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5381.710364] Key type .llcrypt unregistered [ 5381.711308] Key type ._llcrypt unregistered [ 5391.327409] Key type ._llcrypt registered [ 5391.328421] Key type .llcrypt registered [ 5391.367968] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5391.458221] Key type .llcrypt unregistered [ 5391.459396] Key type ._llcrypt unregistered [ 5401.098221] Key type ._llcrypt registered [ 5401.099149] Key type .llcrypt registered [ 5401.132451] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5401.225313] Key type .llcrypt unregistered [ 5401.226231] Key type ._llcrypt unregistered [ 5419.752420] Key type ._llcrypt registered [ 5419.753745] Key type .llcrypt registered [ 5419.784942] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5419.865318] Key type .llcrypt unregistered [ 5419.866258] Key type ._llcrypt unregistered [ 5429.662883] Key type ._llcrypt registered [ 5429.663745] Key type .llcrypt registered [ 5429.697061] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5429.786353] Key type .llcrypt unregistered [ 5429.787311] Key type ._llcrypt unregistered [ 5439.755768] Key type ._llcrypt registered [ 5439.756679] Key type .llcrypt registered [ 5439.787164] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5439.871402] Key type .llcrypt unregistered [ 5439.872278] Key type ._llcrypt unregistered [ 5456.615696] Key type ._llcrypt registered [ 5456.616629] Key type .llcrypt registered [ 5456.648712] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 5462.540388] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5462.823487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5462.863048] alg: No test for adler32 (adler32-zlib) [ 5463.698110] Lustre: Lustre: Build Version: 2.17.50_193_gf215d40 [ 5463.769549] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5465.343088] Key type lgssc registered [ 5465.642596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5474.790365] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5222 sec ========== 03:53:00 (1772441580) [ 5475.317693] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:53:00 (1772441580) === [ 5476.394240] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:53:01 (1772441581) === [ 5487.904208] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5488.772698] Key type lgssc unregistered [ 5488.883225] LNet: 325700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5488.885276] LNetError: 325700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5488.897298] LNet: Removed LNI 192.168.204.147@tcp [ 5489.158824] Key type .llcrypt unregistered [ 5489.159779] Key type ._llcrypt unregistered