[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483302861 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002393] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008490] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014076] *** VALIDATE selinux *** [ 0.023226] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028815] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031133] *** VALIDATE tmpfs *** [ 0.033414] *** VALIDATE proc *** [ 0.035030] *** VALIDATE cgroup *** [ 0.036015] *** VALIDATE cgroup2 *** [ 0.038220] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041040] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045761] debug: unmapping init [mem 0xffffffffa1a59000-0xffffffffa1a60fff] [ 0.047386] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048633] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052010] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056291] rcu: Hierarchical SRCU implementation. [ 0.058661] smp: Bringing up secondary CPUs ... [ 0.059635] x86: Booting SMP configuration: [ 0.060034] .... node #0, CPUs: #1 #2 #3 [ 0.063703] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144031] node 0 deferred pages initialised in 75ms [ 0.147575] devtmpfs: initialized [ 0.149314] x86/mm: Memory block size: 128MB [ 0.153030] gcov: version magic: 0x41383552 [ 0.156409] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160126] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163326] pinctrl core: initialized pinctrl subsystem [ 0.165220] [ 0.165892] ************************************************************* [ 0.169015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171010] ** ** [ 0.173012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176014] ** ** [ 0.179014] ** This means that this kernel is built to expose internal ** [ 0.181012] ** IOMMU data structures, which may compromise security on ** [ 0.184016] ** your system. ** [ 0.186009] ** ** [ 0.188010] ** If you see this message and you are not debugging the ** [ 0.190014] ** kernel, report this immediately to your vendor! ** [ 0.193011] ** ** [ 0.195009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197012] ************************************************************* [ 0.200654] NET: Registered protocol family 16 [ 0.203443] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212109] cpuidle: using governor menu [ 0.213712] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214625] PCI: Using configuration type 1 for base access [ 0.216188] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233116] cryptd: max_cpu_qlen set to 1000 [ 0.237262] ACPI: Added _OSI(Module Device) [ 0.239016] ACPI: Added _OSI(Processor Device) [ 0.241016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.249058] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256329] ACPI: Interpreter enabled [ 0.258091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260014] ACPI: Using IOAPIC for interrupt routing [ 0.262124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276528] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279076] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.286070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291391] acpiphp: Slot [2] registered [ 0.293199] acpiphp: Slot [5] registered [ 0.295223] acpiphp: Slot [6] registered [ 0.297216] acpiphp: Slot [7] registered [ 0.299183] acpiphp: Slot [8] registered [ 0.301164] acpiphp: Slot [9] registered [ 0.303317] acpiphp: Slot [10] registered [ 0.304229] acpiphp: Slot [3] registered [ 0.306148] acpiphp: Slot [4] registered [ 0.308154] acpiphp: Slot [11] registered [ 0.309121] acpiphp: Slot [12] registered [ 0.311119] acpiphp: Slot [13] registered [ 0.313163] acpiphp: Slot [14] registered [ 0.314128] acpiphp: Slot [15] registered [ 0.316143] acpiphp: Slot [16] registered [ 0.317188] acpiphp: Slot [17] registered [ 0.319147] acpiphp: Slot [18] registered [ 0.322144] acpiphp: Slot [19] registered [ 0.323076] acpiphp: Slot [20] registered [ 0.324079] acpiphp: Slot [21] registered [ 0.326093] acpiphp: Slot [22] registered [ 0.328108] acpiphp: Slot [23] registered [ 0.329092] acpiphp: Slot [24] registered [ 0.331119] acpiphp: Slot [25] registered [ 0.333155] acpiphp: Slot [26] registered [ 0.335123] acpiphp: Slot [27] registered [ 0.337109] acpiphp: Slot [28] registered [ 0.339125] acpiphp: Slot [29] registered [ 0.340097] acpiphp: Slot [30] registered [ 0.342101] acpiphp: Slot [31] registered [ 0.343059] PCI host bridge to bus 0000:00 [ 0.345022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.347025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.354021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.356036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.359029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.363181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365954] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368729] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.381000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.384059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.397638] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.400820] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.404052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.407735] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.412017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426901] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.432019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.437594] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.455031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.485944] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.491951] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.523019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.534635] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.541015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.547016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.573840] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.579016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.586015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.600020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.610278] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.653353] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.666015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.680041] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.692166] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.695370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.697474] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.700404] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.708085] iommu: Default domain type: Passthrough [ 0.710496] SCSI subsystem initialized [ 0.712182] ACPI: bus type USB registered [ 0.714145] usbcore: registered new interface driver usbfs [ 0.716108] usbcore: registered new interface driver hub [ 0.719095] usbcore: registered new device driver usb [ 0.721220] pps_core: LinuxPPS API ver. 1 registered [ 0.723013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727063] PTP clock support registered [ 0.730091] EDAC MC: Ver: 3.0.0 [ 0.731796] PCI: Using ACPI for IRQ routing [ 0.733958] NetLabel: Initializing [ 0.736012] NetLabel: domain hash size = 128 [ 0.738009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.740079] NetLabel: unlabeled traffic allowed by default [ 0.742096] vgaarb: loaded [ 0.743267] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.746025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.753009] clocksource: Switched to clocksource kvm-clock [ 0.861628] VFS: Disk quotas dquot_6.6.0 [ 0.863560] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.866524] *** VALIDATE ramfs *** [ 0.868029] *** VALIDATE hugetlbfs *** [ 0.870179] pnp: PnP ACPI init [ 0.873386] pnp: PnP ACPI: found 6 devices [ 0.890823] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.894872] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.897599] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.900251] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.902880] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.905762] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.908797] NET: Registered protocol family 2 [ 0.911411] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.916744] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.920582] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.925988] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.930075] TCP: Hash tables configured (established 65536 bind 65536) [ 0.933087] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.936402] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.940032] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942385] NET: Registered protocol family 1 [ 0.945074] RPC: Registered named UNIX socket transport module. [ 0.947715] RPC: Registered udp transport module. [ 0.950332] RPC: Registered tcp transport module. [ 0.952348] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.955226] NET: Registered protocol family 44 [ 0.957141] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.959674] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.961384] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.963266] PCI: CLS 0 bytes, default 64 [ 0.965191] Unpacking initramfs... [ 2.349977] debug: unmapping init [mem 0xffff95907cc54000-0xffff95907ffbffff] [ 2.354185] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.356785] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.360350] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.840245] Initialise system trusted keyrings [ 2.842431] Key type blacklist registered [ 2.844634] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.854092] zbud: loaded [ 2.857432] *** VALIDATE nfs *** [ 2.858799] *** VALIDATE nfs4 *** [ 2.860518] pstore: using deflate compression [ 2.864118] Platform Keyring initialized [ 2.956549] NET: Registered protocol family 38 [ 2.958700] Key type asymmetric registered [ 2.960406] Asymmetric key parser 'x509' registered [ 2.962538] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.965950] io scheduler mq-deadline registered [ 2.967877] io scheduler kyber registered [ 2.969790] io scheduler bfq registered [ 2.971946] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.974835] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.977344] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.979790] ACPI: Power Button [PWRF] [ 2.985207] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.992615] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.006955] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.014438] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.030313] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.057938] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.085179] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.089818] Non-volatile memory driver v1.3 [ 3.091831] Linux agpgart interface v0.103 [ 3.121814] virtio_blk virtio1: [vda] 135184 512-byte logical blocks (69.2 MB/66.0 MiB) [ 3.125270] vda: detected capacity change from 0 to 69214208 [ 3.143165] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.146940] vdb: detected capacity change from 0 to 1073741824 [ 3.161768] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.165825] vdc: detected capacity change from 0 to 2621440000 [ 3.180317] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.183366] vdd: detected capacity change from 0 to 2621440000 [ 3.200062] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.203690] vde: detected capacity change from 0 to 4294967296 [ 3.219231] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.221921] vdf: detected capacity change from 0 to 4294967296 [ 3.229706] libphy: Fixed MDIO Bus: probed [ 3.235156] usbcore: registered new interface driver usbserial_generic [ 3.238144] usbserial: USB Serial support registered for generic [ 3.241218] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.246590] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.249043] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.251700] mousedev: PS/2 mouse device common for all mice [ 3.254839] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.256652] rtc_cmos 00:05: RTC can wake from S4 [ 3.262743] rtc_cmos 00:05: registered as rtc0 [ 3.262831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.265284] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.272363] intel_pstate: CPU model not supported [ 3.274986] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.280372] hid: raw HID events driver (C) Jiri Kosina [ 3.282982] usbcore: registered new interface driver usbhid [ 3.285372] usbhid: USB HID core driver [ 3.287282] drop_monitor: Initializing network drop monitor service [ 3.290126] Initializing XFRM netlink socket [ 3.292244] NET: Registered protocol family 10 [ 3.295372] Segment Routing with IPv6 [ 3.296641] NET: Registered protocol family 17 [ 3.298779] mpls_gso: MPLS GSO support [ 3.304245] RAS: Correctable Errors collector initialized. [ 3.306335] AVX version of gcm_enc/dec engaged. [ 3.307630] AES CTR mode by8 optimization enabled [ 3.378885] sched_clock: Marking stable (3378861493, 0)->(4334639072, -955777579) [ 3.383150] registered taskstats version 1 [ 3.384951] Loading compiled-in X.509 certificates [ 3.386990] zswap: loaded using pool lzo/zbud [ 3.407835] Key type big_key registered [ 3.420594] Key type encrypted registered [ 3.422551] ima: No TPM chip found, activating TPM-bypass! [ 3.424832] ima: Allocated hash algorithm: sha1 [ 3.427038] ima: No architecture policies found [ 3.428984] evm: Initialising EVM extended attributes: [ 3.431146] evm: security.selinux [ 3.432556] evm: security.ima [ 3.433720] evm: security.capability [ 3.435209] evm: HMAC attrs: 0x1 [ 3.437686] rtc_cmos 00:05: setting system clock to 2026-03-30 16:05:09 UTC (1774886709) [ 3.444826] debug: unmapping init [mem 0xffffffffa2a03000-0xffffffffa2bfffff] [ 3.448614] debug: unmapping init [mem 0xffffffffa1782000-0xffffffffa1a58fff] [ 3.458075] Write protecting the kernel read-only data: 28672k [ 3.461266] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 3.463891] debug: unmapping init [mem 0xffffffffa0714000-0xffffffffa07fffff] [ 3.495252] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.504139] systemd[1]: Detected virtualization kvm. [ 3.506251] systemd[1]: Detected architecture x86-64. [ 3.508433] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.537094] systemd[1]: No hostname configured. [ 3.538534] systemd[1]: Set hostname to . [ 3.540078] random: systemd: uninitialized urandom read (16 bytes read) [ 3.541873] systemd[1]: Initializing machine ID from random generator. [ 3.587231] random: ln: uninitialized urandom read (6 bytes read) [ 3.659140] random: systemd: uninitialized urandom read (16 bytes read) [ 3.662165] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.669795] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.674111] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [[ 3.843396] random: fast init done  OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.230579] device-mapper: uevent: version 1.0.3 [ 4.235113] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.909438] virtio_net virtio0 ens2: renamed from eth0 [ 5.060640] scsi host0: ata_piix [ 5.073808] scsi host1: ata_piix [ 5.076352] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.079711] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.594132] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.969396] random: crng init done [ 9.976497] random: 7 urandom warning(s) missed due to ratelimiting [ 10.477575] hrtimer: interrupt took 5518063 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.645471] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.375249] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.344779] SELinux: Disabled at runtime. [ 15.496084] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.513902] systemd[1]: Detected virtualization kvm. [ 15.525787] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.658500] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.669744] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.702562] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.711852] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.719254] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.736121] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.810349] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 18.429685] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.660540] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.031679] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.131258] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.730710] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.837672] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 27.146616] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 27.848107] NFS: Registering the id_resolver key type [ 27.857291] Key type id_resolver registered [ 27.862534] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg413-server login: [ 82.096300] libcfs: loading out-of-tree module taints kernel. [ 82.168302] Key type ._llcrypt registered [ 82.170994] Key type .llcrypt registered [ 82.325982] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 101.270651] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 102.651022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.679135] alg: No test for adler32 (adler32-zlib) [ 104.170340] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 104.914178] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 106.641980] Key type lgssc registered [ 108.148852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.073519] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.798197] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 182.073991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.144557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.568114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.611382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.794953] Lustre: lustre-MDT0000: new disk, initializing [ 183.961190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.019351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.731767] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.118207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.306833] Lustre: 6499:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 206.348964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 206.352401] Lustre: Skipped 1 previous similar message [ 206.488395] Lustre: lustre-MDT0001: new disk, initializing [ 206.589524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.632973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 206.646683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.502865] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.764768] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.203183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.622404] Lustre: lustre-OST0000: new disk, initializing [ 228.627260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.704688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.085451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.092908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.212061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.206595] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.854882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.039897] Lustre: lustre-OST0001: new disk, initializing [ 253.049066] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 253.134463] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.922656] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.159686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.178514] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.229981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 272.126794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.501452] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.561971] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.225721] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing yml_node [ 303.422131] Lustre: DEBUG MARKER: Client: 2.17.50.165 [ 306.610846] Lustre: DEBUG MARKER: MDS: 2.17.50.165 [ 309.977682] Lustre: DEBUG MARKER: OSS: 2.17.50.165 [ 312.420401] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 30 12:10:16 EDT 2026 [ 332.471736] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 334.042413] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.232321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.236022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.250084] Lustre: Skipped 2 previous similar messages [ 352.255296] Lustre: Skipped 1 previous similar message [ 356.756523] Lustre: server umount lustre-MDT0000 complete [ 361.017895] LustreError: 6490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887067 with bad export cookie 17577764742641891162 [ 361.020967] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.035850] LustreError: 6490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 362.464066] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.469337] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.472635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.472645] Lustre: Skipped 2 previous similar messages [ 362.499143] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 362.545295] Lustre: Skipped 2 previous similar messages [ 366.498671] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.500876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.544654] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 366.570734] Lustre: Skipped 2 previous similar messages [ 367.459778] Lustre: server umount lustre-MDT0001 complete [ 371.706498] Lustre: server umount lustre-OST0000 complete [ 375.884822] Lustre: server umount lustre-OST0001 complete [ 382.392957] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 391.096557] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 433.312229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.694353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 433.761923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 433.915391] Lustre: lustre-MDT0000: new disk, initializing [ 434.066939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.103143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 439.015278] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 450.663425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.815638] Lustre: 14979:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.889502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.897590] Lustre: Skipped 1 previous similar message [ 451.073897] Lustre: lustre-MDT0001: new disk, initializing [ 451.320683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 451.381534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 451.399888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 455.852878] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 463.439378] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 466.977458] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 474.207994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.619505] Lustre: lustre-OST0000: new disk, initializing [ 474.627578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 474.745981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.373266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.387415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.569652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.255306] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 489.592861] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 494.012411] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.201157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 498.390632] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.625811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.241953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.247135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.251721] LustreError: Skipped 1 previous similar message [ 502.255722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.279104] Lustre: Skipped 1 previous similar message [ 502.298932] Lustre: Skipped 1 previous similar message [ 506.699171] Lustre: server umount lustre-OST0000 complete [ 512.481121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.491842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.506340] Lustre: Skipped 1 previous similar message [ 512.523882] Lustre: Skipped 1 previous similar message [ 516.931481] Lustre: server umount lustre-MDT0000 complete [ 517.612917] LustreError: 14986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.641107] LustreError: 14986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 520.878177] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887226 with bad export cookie 17577764742641892688 [ 520.878915] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.895313] LustreError: 14969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 521.385752] Lustre: server umount lustre-MDT0001 complete [ 528.817240] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:13:53 (1774887233) [ 550.111203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.822270] LustreError: 19450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.951513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.718167] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 556.004987] LustreError: 19450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.391660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.857050] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 576.567172] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 580.898984] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 583.116335] Lustre: Failing over lustre-MDT0000 [ 583.322755] Lustre: server umount lustre-MDT0000 complete [ 585.188146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.189532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.195460] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.195474] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 585.245565] Lustre: Skipped 1 previous similar message [ 601.568281] Lustre: 3650:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774887291/real 1774887291] req@ffff9590fe9e2a00 x1861103711844992/t0(0) o400->MGC192.168.204.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774887307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 601.575689] LustreError: 20160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.609168] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.657644] LustreError: 20160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 602.893473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.356303] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 616.999565] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 617.441458] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 617.450154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 617.468746] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.009484] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 632.801565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.802897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.824203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.828372] Lustre: Skipped 1 previous similar message [ 632.850200] Lustre: Skipped 1 previous similar message [ 635.169281] Lustre: server umount lustre-MDT0000 complete [ 636.962403] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.994308] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 639.505279] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887345 with bad export cookie 17577764742641894109 [ 639.511890] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.522925] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 640.218287] Lustre: server umount lustre-MDT0001 complete [ 650.610226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.561707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.567763] Lustre: Skipped 2 previous similar messages [ 656.384679] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 666.049898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.332657] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 677.716279] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 681.683184] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 683.895727] Lustre: Failing over lustre-MDT0000 [ 684.275036] Lustre: server umount lustre-MDT0000 complete [ 687.073362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 687.076372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.110976] Lustre: Skipped 1 previous similar message [ 701.424589] LustreError: 22443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.455304] LustreError: 22443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 703.461675] Lustre: 3649:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774887393/real 1774887393] req@ffff9590c947bb80 x1861103711873280/t0(0) o400->MGC192.168.204.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774887409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 703.501558] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.728662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.696124] LustreError: 3648:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9591013a8700 x1861103711876096/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 714.295109] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 719.331307] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 719.337533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 719.353540] Lustre: Skipped 1 previous similar message [ 719.379167] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 719.790186] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 727.346588] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 731.700444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.702535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 731.713277] Lustre: Skipped 1 previous similar message [ 731.732668] Lustre: Skipped 1 previous similar message [ 738.156783] Lustre: server umount lustre-MDT0000 complete [ 742.490545] LustreError: 22429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887448 with bad export cookie 17577764742641895047 [ 742.501045] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.504354] LustreError: 22429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 743.015639] Lustre: server umount lustre-MDT0001 complete [ 752.967242] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:17:37 (1774887457) [ 771.450475] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 781.805353] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 824.129621] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 837.743040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.149592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 838.188375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 838.265505] Lustre: lustre-MDT0000: new disk, initializing [ 838.398451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.403351] Lustre: Skipped 2 previous similar messages [ 838.469184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 843.414560] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 856.959622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.127649] Lustre: 28434:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 857.534074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 863.060592] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 871.844297] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.093298] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 883.860788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 885.670644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 885.737787] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 885.840287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 885.855258] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 889.934812] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 898.258770] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 904.260269] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 931.812102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.823434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.838981] LustreError: Skipped 1 previous similar message [ 931.844594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.851952] Lustre: Skipped 1 previous similar message [ 931.889928] Lustre: Skipped 5 previous similar messages [ 937.249928] Lustre: server umount lustre-OST0000 complete [ 955.359600] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 955.708000] Lustre: server umount lustre-MDT0000 complete [ 957.425793] LustreError: 28442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.443296] LustreError: 28442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 959.456608] LustreError: 28426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887665 with bad export cookie 17577764742641895453 [ 959.458995] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 959.495868] LustreError: 28426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 959.960920] Lustre: server umount lustre-MDT0001 complete [ 970.858365] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 973.973387] Key type lgssc unregistered [ 974.354877] LNet: 32027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.360652] LNetError: 32027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.382201] LNet: Removed LNI 192.168.204.113@tcp [ 975.392172] Key type .llcrypt unregistered [ 975.401228] Key type ._llcrypt unregistered [ 1004.116183] Key type ._llcrypt registered [ 1004.118910] Key type .llcrypt registered [ 1004.233697] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1005.702048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.744397] alg: No test for adler32 (adler32-zlib) [ 1006.803151] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1007.068812] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1008.783393] Key type lgssc registered [ 1009.899861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1018.653209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1018.705612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.388807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.959825] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1030.629152] LustreError: 33425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.656276] LustreError: 33425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1035.771750] LustreError: 33425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.706381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.078314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1042.192314] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1048.463404] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1053.128142] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1061.295103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.815338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.846540] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1065.941615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1068.064710] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1073.980901] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1089.637621] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:23:13 (1774887793) [ 1092.415272] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1094.795082] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1102.510521] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:23:27 (1774887807) [ 1111.646909] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:23:36 (1774887816) [ 1113.056460] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1123.730797] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1123.742185] Lustre: Skipped 2 previous similar messages [ 1126.110605] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:23:50 (1774887830) [ 1128.924392] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1128.935349] Lustre: Skipped 2 previous similar messages [ 1137.020644] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1137.030721] Lustre: Skipped 1 previous similar message [ 1153.246621] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1153.253293] Lustre: Skipped 5 previous similar messages [ 1156.499200] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:24:20 (1774887860) [ 1159.480070] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1163.566909] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1172.758837] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:24:37 (1774887877) [ 1191.048333] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1191.056211] Lustre: Skipped 7 previous similar messages [ 1194.594352] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:24:57 (1774887897) [ 1216.481226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1216.492265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.535295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1219.555364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1219.566527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.610701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1224.673569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1224.689940] Lustre: Skipped 2 previous similar messages [ 1229.279146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.414648] Lustre: server umount lustre-OST0000 complete [ 1234.917483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.919262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.947908] Lustre: Skipped 1 previous similar message [ 1239.579952] Lustre: server umount lustre-MDT0000 complete [ 1240.033972] LustreError: 33426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.057458] LustreError: 33426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1243.169842] LustreError: 33405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887949 with bad export cookie 8277594965581029059 [ 1243.176195] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.186128] LustreError: 33405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1243.719567] Lustre: server umount lustre-MDT0001 complete [ 1254.835688] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1257.913439] Key type lgssc unregistered [ 1258.248701] LNet: 40932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.260605] LNetError: 40932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.272914] LNet: Removed LNI 192.168.204.113@tcp [ 1259.182252] Key type .llcrypt unregistered [ 1259.191557] Key type ._llcrypt unregistered [ 1278.705224] Key type ._llcrypt registered [ 1278.719238] Key type .llcrypt registered [ 1278.838802] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 1295.539982] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1296.909468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.221459] alg: No test for adler32 (adler32-zlib) [ 1298.386235] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1298.664714] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1300.375143] Key type lgssc registered [ 1301.829754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.471934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.529404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.008693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1344.079849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1344.207637] Lustre: lustre-MDT0000: new disk, initializing [ 1344.336767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.363947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1349.042802] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1362.569223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.737192] Lustre: 44054:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1362.783158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1362.787237] Lustre: Skipped 1 previous similar message [ 1362.861473] Lustre: lustre-MDT0001: new disk, initializing [ 1362.994861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1363.053349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1363.067109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1368.794370] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1376.869780] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1381.282264] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1387.822413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1388.283327] Lustre: lustre-OST0000: new disk, initializing [ 1388.297672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1388.455679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.175633] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1395.856260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1395.884379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1396.094183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1402.838458] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1407.912887] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1408.193252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1412.572673] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1412.808548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1416.675410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.676155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.689581] LustreError: Skipped 1 previous similar message [ 1416.693520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.711550] Lustre: Skipped 1 previous similar message [ 1416.740867] Lustre: Skipped 1 previous similar message [ 1421.799321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.815337] Lustre: Skipped 1 previous similar message [ 1426.927634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.938935] Lustre: Skipped 1 previous similar message [ 1428.959128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1429.276391] Lustre: server umount lustre-OST0000 complete [ 1434.597839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1434.610840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.625095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.160987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.537926] Lustre: server umount lustre-MDT0000 complete [ 1444.475190] LustreError: 44047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888150 with bad export cookie 3022041720597478790 [ 1444.478340] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1444.501487] LustreError: 44047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1444.974343] Lustre: server umount lustre-MDT0001 complete [ 1455.801705] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:29:20 (1774888160) [ 1463.045171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.637897] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.654083] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1463.705442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.201952] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1468.917868] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.996751] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.114248] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.303901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.622723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1484.167931] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1491.489910] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1496.067558] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1503.723442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1504.176699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.441757] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1516.469488] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1529.582646] LustreError: 50327:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1530.542061] Lustre: Modifying parameter general.timeout=1 in log params [ 1531.612658] Lustre: Modifying parameter general.timeout=2 in log params [ 1534.304593] Lustre: Modifying parameter general.timeout=5 in log params [ 1534.315320] Lustre: Skipped 2 previous similar messages [ 1538.477217] Lustre: Modifying parameter general.timeout=10 in log params [ 1538.487482] Lustre: Skipped 4 previous similar messages [ 1546.984497] Lustre: Modifying parameter general.timeout=21 in log params [ 1546.988212] Lustre: Skipped 10 previous similar messages [ 1563.619956] Lustre: Modifying parameter general.timeout=38 in log params [ 1563.624318] Lustre: Skipped 16 previous similar messages [ 1596.165106] Lustre: Modifying parameter general.timeout=73 in log params [ 1596.175545] Lustre: Skipped 34 previous similar messages [ 1662.276355] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:32:46 (1774888366) [ 1663.234474] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1663.241379] Lustre: Skipped 57 previous similar messages [ 1665.272152] Lustre: 57277:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1667.037093] Lustre: 57376:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1670.775492] Lustre: 57572:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1670.789716] Lustre: 57572:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1675.046127] Lustre: 57788:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1675.058834] Lustre: 57788:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1703.373504] Lustre: 59151:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1703.410634] LustreError: 59151:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1708.157762] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:33:32 (1774888412) [ 1723.364824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.368301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.391549] Lustre: Skipped 1 previous similar message [ 1723.417253] Lustre: Skipped 4 previous similar messages [ 1726.929279] Lustre: server umount lustre-MDT0000 complete [ 1728.481136] LustreError: 48107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.504431] LustreError: 48107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1731.130448] LustreError: 48091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888437 with bad export cookie 3022041720597479672 [ 1731.138434] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1731.148984] LustreError: 48091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1731.572804] Lustre: server umount lustre-MDT0001 complete [ 1746.113919] Lustre: server umount lustre-OST0000 complete [ 1775.188654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.556411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1775.949557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.436378] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1792.290620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.438706] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1792.542674] Lustre: 61680:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.801706] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1805.066442] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1809.572872] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1816.918719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1817.208425] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1817.613456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.618109] Lustre: Skipped 1 previous similar message [ 1824.355100] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1830.487116] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1832.699516] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1832.712211] Lustre: Skipped 22 previous similar messages [ 1832.792934] Lustre: 63910:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1843.170971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1843.178579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.191087] Lustre: Skipped 1 previous similar message [ 1843.199199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.125297] Lustre: server umount lustre-OST0000 complete [ 1849.312260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1849.323623] LustreError: Skipped 1 previous similar message [ 1853.933187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.946401] Lustre: Skipped 2 previous similar messages [ 1855.470213] Lustre: server umount lustre-MDT0000 complete [ 1859.041526] LustreError: 61693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.079573] LustreError: 61693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1860.108641] LustreError: 62418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888566 with bad export cookie 3022041720597482717 [ 1860.110226] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1860.119985] LustreError: 62418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1860.560965] Lustre: server umount lustre-MDT0001 complete [ 1872.506346] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1875.755501] Key type lgssc unregistered [ 1876.122475] LNet: 65103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.127556] LNetError: 65103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.142488] LNet: Removed LNI 192.168.204.113@tcp [ 1877.151191] Key type .llcrypt unregistered [ 1877.156043] Key type ._llcrypt unregistered [ 1904.325257] Key type ._llcrypt registered [ 1904.329386] Key type .llcrypt registered [ 1904.510184] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1906.189353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1906.239861] alg: No test for adler32 (adler32-zlib) [ 1907.444163] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 1907.738883] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1909.511239] Key type lgssc registered [ 1910.814578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.262049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.293914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.776987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.029191] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1930.211098] LustreError: 66499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.231145] LustreError: 66499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1934.297202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.008147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.727630] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1946.715305] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1950.830540] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1957.858060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.650669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.939629] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1971.886612] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1980.780725] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1991.649134] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.651722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.670357] Lustre: Skipped 1 previous similar message [ 1991.692128] Lustre: Skipped 1 previous similar message [ 1996.256712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1996.267527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.287930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.849266] Lustre: server umount lustre-MDT0000 complete [ 2001.893297] LustreError: 66499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.916549] LustreError: 66499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2002.697883] LustreError: 66479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888708 with bad export cookie 167680141597325936 [ 2002.707366] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.710209] LustreError: 66479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2003.026677] Lustre: server umount lustre-MDT0001 complete [ 2017.938612] Lustre: server umount lustre-OST0000 complete [ 2049.422937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.055420] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2050.128961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2050.725621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.595813] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2067.383871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.473405] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2067.507762] Lustre: 71038:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2068.289794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.075751] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2078.924237] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2083.097807] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2090.298298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.509251] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2090.898309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.294197] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2103.751270] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2105.265203] Lustre: 73275:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2107.026649] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2115.044618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.048924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.050418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2115.050426] Lustre: Skipped 2 previous similar messages [ 2115.055867] LustreError: Skipped 1 previous similar message [ 2115.090840] Lustre: Skipped 1 previous similar message [ 2118.891133] Lustre: server umount lustre-OST0000 complete [ 2124.260204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2124.279856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.290888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.295865] Lustre: Skipped 1 previous similar message [ 2129.243921] Lustre: server umount lustre-MDT0000 complete [ 2130.411556] LustreError: 71046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.452414] LustreError: 71046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2133.621175] LustreError: 71031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888839 with bad export cookie 167680141597326776 [ 2133.629423] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2134.181170] Lustre: server umount lustre-MDT0001 complete [ 2144.759907] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2148.660273] Key type lgssc unregistered [ 2149.184257] LNet: 74469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.193461] LNetError: 74469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.215780] LNet: Removed LNI 192.168.204.113@tcp [ 2150.544357] Key type .llcrypt unregistered [ 2150.546352] Key type ._llcrypt unregistered [ 2180.326627] Key type ._llcrypt registered [ 2180.329097] Key type .llcrypt registered [ 2180.431627] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2181.816054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.846072] alg: No test for adler32 (adler32-zlib) [ 2182.786212] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2182.953237] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2184.743247] Key type lgssc registered [ 2186.524493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2195.265765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2195.283247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.857311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.877399] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2206.178762] LustreError: 75867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.198062] LustreError: 75867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2211.273697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.304751] LustreError: 75867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.235824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2218.195256] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2225.915762] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2230.809477] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2240.288715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2240.888311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.050931] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2254.087841] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2260.833185] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2261.788834] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2263.663843] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2263.669698] Lustre: Skipped 1 previous similar message [ 2266.579166] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2266.583709] Lustre: Skipped 2 previous similar messages [ 2271.338130] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2271.344094] Lustre: Skipped 4 previous similar messages [ 2279.991979] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.998745] Lustre: Skipped 8 previous similar messages [ 2296.223624] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2296.231110] Lustre: Skipped 15 previous similar messages [ 2328.626215] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2328.633509] Lustre: Skipped 35 previous similar messages [ 2363.205419] Lustre: 83636:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2366.034687] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:44:30 (1774889070) [ 2367.593837] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2369.371751] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:44:34 (1774889074) [ 2375.137533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.159740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.185524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2379.233073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2379.236067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.237145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2380.509662] Lustre: server umount lustre-OST0000 complete [ 2384.352753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.373056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.376607] Lustre: Skipped 2 previous similar messages [ 2389.475993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.481811] Lustre: Skipped 1 previous similar message [ 2390.452617] Lustre: server umount lustre-MDT0000 complete [ 2393.984510] LustreError: 78913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889100 with bad export cookie 12695797527907825298 [ 2394.002402] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2394.348628] Lustre: server umount lustre-MDT0001 complete [ 2405.387766] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2408.342961] Key type lgssc unregistered [ 2408.623870] LNet: 84869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.631298] LNetError: 84869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.655220] LNet: Removed LNI 192.168.204.113@tcp [ 2409.617861] Key type .llcrypt unregistered [ 2409.620522] Key type ._llcrypt unregistered [ 2413.910578] Key type ._llcrypt registered [ 2413.913287] Key type .llcrypt registered [ 2414.015823] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2418.103603] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules [ 2419.151212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.276531] alg: No test for adler32 (adler32-zlib) [ 2420.348017] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2420.551966] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2422.225243] Key type lgssc registered [ 2423.561064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2424.859685] LustreError: 86037:0:(lu_object.c:1434:lu_context_key_register()) cfs_race id 60d sleeping [ 2441.690812] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2446.368666] LustreError: 86037:0:(lu_object.c:1434:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2451.989602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2452.022844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.840235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.494781] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2463.202274] LustreError: 87044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.225468] LustreError: 87044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2467.984562] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:46:12 (1774889172) [ 2468.320487] LustreError: 87044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.124202] LustreError: 87075:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2473.132728] LustreError: 87075:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2473.140120] LustreError: 87075:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2473.533138] Lustre: server umount lustre-MDT0000 complete [ 2489.393975] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2492.054067] Key type lgssc unregistered [ 2492.388169] LNet: 88536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.396641] LNetError: 88536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.425803] LNet: Removed LNI 192.168.204.113@tcp [ 2493.181289] Key type .llcrypt unregistered [ 2493.183435] Key type ._llcrypt unregistered [ 2514.204786] Key type ._llcrypt registered [ 2514.207772] Key type .llcrypt registered [ 2514.317598] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2515.299211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2515.335240] alg: No test for adler32 (adler32-zlib) [ 2516.508376] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2516.824831] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2518.551685] Key type lgssc registered [ 2519.761417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2527.305263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2527.330680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.914714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.321752] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2538.477289] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.507780] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2542.300755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.104701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2547.839271] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2554.283894] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2558.135848] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2564.540753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2565.111874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2570.389022] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2575.676740] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2621.255408] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:48:46 (1774889326) [ 2622.752658] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2624.837894] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:48:49 (1774889329) [ 2629.603988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.615885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2629.615966] Lustre: Skipped 1 previous similar message [ 2629.616540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.724427] LustreError: 89738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.743158] LustreError: 89738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2634.898547] Lustre: server umount lustre-MDT0000 complete [ 2638.852852] LustreError: 89718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889344 with bad export cookie 18032535034557435515 [ 2638.870749] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.113419] Lustre: server umount lustre-MDT0001 complete [ 2653.229280] Lustre: server umount lustre-OST0000 complete [ 2663.910135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.503506] LustreError: 93676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.678648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.517692] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2670.048872] LustreError: 93676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.148704] LustreError: 93676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.227235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.858910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2680.784083] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2686.089328] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2689.718608] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2701.591531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.744128] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2701.749167] LustreError: 93663:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2701.769659] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2701.782241] LustreError: 95535:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2701.788291] LustreError: 95535:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2701.794080] LustreError: 95544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.796039] LustreError: 95535:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2701.871649] Lustre: server umount lustre-OST0000 complete [ 2701.874640] LustreError: 95535:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2709.759699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.966227] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2709.974993] LustreError: 93663:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2709.993196] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2710.002260] LustreError: 96000:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2710.008447] LustreError: 96000:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2710.014462] LustreError: 96000:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2710.101108] Lustre: server umount lustre-OST0000 complete [ 2710.103282] LustreError: 96000:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2718.325849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.521861] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2718.526269] Lustre: Found index 0 for lustre-OST0000, updating log [ 2718.531893] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2718.621875] Lustre: lustre-OST0000: new disk, initializing [ 2718.629424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2718.928447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.340520] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2728.656818] Lustre: Failing over lustre-OST0000 [ 2728.830366] Lustre: server umount lustre-OST0000 complete [ 2733.027387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2733.038057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.054662] Lustre: Skipped 1 previous similar message [ 2733.063101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.067643] Lustre: Skipped 2 previous similar messages [ 2737.128209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.129987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.659350] Lustre: server umount lustre-MDT0000 complete [ 2741.290540] LustreError: 93676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.303341] LustreError: 93676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2742.447366] LustreError: 93656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889448 with bad export cookie 18032535034557438252 [ 2742.456192] LustreError: 93656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2742.459958] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.701489] Lustre: server umount lustre-MDT0001 complete [ 2750.581480] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:50:55 (1774889455) [ 2764.880292] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2775.019488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.266550] LustreError: 98848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.331220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.818516] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2786.854798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.113677] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2793.573198] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2799.110626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.354342] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2809.804216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.873952] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2809.957337] Lustre: lustre-OST0001: new disk, initializing [ 2809.961272] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2810.181590] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.184823] Lustre: Skipped 2 previous similar messages [ 2811.121691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2811.136075] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2811.264493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2814.964723] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2821.095911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2826.095883] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2837.495433] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [ 2837.501112] Lustre: Skipped 1 previous similar message [ 2837.990426] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.001608] Lustre: Skipped 2 previous similar messages [ 2840.284862] Lustre: server umount lustre-MDT0001 complete [ 2842.592782] LustreError: 98843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.603446] LustreError: 98843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2846.118859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.209525] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2846.212193] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2846.240740] Lustre: 98834:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2847.472361] LustreError: 102690:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2847.481814] LustreError: 102690:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2847.487340] LustreError: 98839:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2847.540244] LustreError: 102695:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.113@tcp: cfg command failed: rc = -17 [ 2847.546615] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2847.546615] [ 2849.064062] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2851.818921] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2855.324582] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 12:52:40 (1774889560) [ 2865.781580] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2878.123060] Lustre: 104682:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2893.966572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2896.615838] Lustre: 105814:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2979.816145] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.816989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2979.817690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2979.817697] Lustre: Skipped 3 previous similar messages [ 2979.829961] Lustre: Skipped 1 previous similar message [ 2984.484868] Lustre: server umount lustre-MDT0000 complete [ 2984.936167] LustreError: 98842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.948511] LustreError: 98842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2987.189910] LustreError: 99956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889693 with bad export cookie 18032535034557439253 [ 2987.195532] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2987.197125] LustreError: 99956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2987.404689] Lustre: server umount lustre-MDT0001 complete [ 3000.489341] Lustre: server umount lustre-OST0000 complete [ 3013.911463] Lustre: server umount lustre-OST0001 complete [ 3018.852090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3026.749480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3032.985798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3040.485934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3054.681425] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3062.052303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.066571] Lustre: lustre-MDT0000: reset Object Index mappings [ 3062.529618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.534231] Lustre: Skipped 1 previous similar message [ 3065.434749] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3072.045796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.062196] Lustre: lustre-MDT0001: reset Object Index mappings [ 3076.021880] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3078.540331] Lustre: 110864:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3084.397936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.157884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3089.754142] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3096.673924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3098.174440] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3098.187465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3098.203130] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3102.048752] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3108.506917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3111.505756] Lustre: 112715:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3162.301547] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 12:57:47 (1774889867) [ 3190.245478] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.248174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.259743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3190.267488] Lustre: Skipped 5 previous similar messages [ 3190.286040] Lustre: Skipped 6 previous similar messages [ 3195.568041] Lustre: server umount lustre-MDT0000 complete [ 3198.401074] LustreError: 109732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889904 with bad export cookie 18032535034557486146 [ 3198.407975] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3198.414233] LustreError: 109732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3204.928485] Lustre: server umount lustre-MDT0001 complete [ 3214.330163] Lustre: server umount lustre-OST0000 complete [ 3223.588831] Lustre: server umount lustre-OST0001 complete [ 3228.796312] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3236.862570] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3263.031358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.240800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3263.273439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3263.328920] Lustre: lustre-MDT0000: new disk, initializing [ 3263.417619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.422692] Lustre: Skipped 3 previous similar messages [ 3263.438170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3266.363781] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3271.937613] Lustre: server umount lustre-MDT0000 complete [ 3278.531544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.830934] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3278.853108] Lustre: 116930:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3278.868427] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3282.263705] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3286.069901] Lustre: server umount lustre-MDT0000 complete [ 3292.907701] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 12:59:58 (1774889998) [ 3294.099309] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3295.297656] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:00:00 (1774890000) [ 3307.790407] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3314.947353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.790418] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3324.407691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.514488] Lustre: 118946:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3324.561517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3324.565546] Lustre: Skipped 1 previous similar message [ 3324.632520] Lustre: lustre-MDT0001: new disk, initializing [ 3324.750763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3324.763260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3327.991062] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3332.127323] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3337.199595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.468793] Lustre: lustre-OST0000: new disk, initializing [ 3337.480150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3338.515439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3338.522371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3338.639982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3341.897293] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3350.813644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.957843] Lustre: lustre-OST0001: new disk, initializing [ 3350.962810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3352.195919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3352.206884] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3352.278472] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3356.144878] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3364.653221] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3367.753197] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3667.892788] Lustre: 127737:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3667.897668] Lustre: 127737:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3667.901805] Lustre: 127737:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3667.905620] Lustre: 127737:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3667.909701] Lustre: 127737:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3667.914565] Lustre: 127737:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3881.877491] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:09:46 (1774890586) [ 3895.266800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.276612] Lustre: Skipped 2 previous similar messages [ 3895.281672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.291252] Lustre: Skipped 11 previous similar messages [ 3900.388447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3900.396385] Lustre: Skipped 3 previous similar messages [ 3900.667531] Lustre: server umount lustre-MDT0000 complete [ 3902.587250] LustreError: 118939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890608 with bad export cookie 18032535034557553353 [ 3902.592326] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.594393] LustreError: 118939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3905.506813] LustreError: 118954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.507298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.515536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3905.524852] LustreError: 118954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3905.540702] Lustre: Skipped 3 previous similar messages [ 3908.963163] Lustre: server umount lustre-MDT0001 complete [ 3917.465431] Lustre: server umount lustre-OST0000 complete [ 3925.499451] Lustre: server umount lustre-OST0001 complete [ 3928.532368] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3933.513404] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3955.147975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.312679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3955.343712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3955.399981] Lustre: lustre-MDT0000: new disk, initializing [ 3955.512206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.521232] Lustre: Skipped 5 previous similar messages [ 3955.538971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3957.914373] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3964.176631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.242388] Lustre: 130738:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3964.399116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3966.581595] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3971.424773] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3973.291186] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3976.923915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3978.953668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3978.960473] Lustre: Skipped 1 previous similar message [ 3978.963484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3979.060284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.242310] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3984.720304] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3989.661144] Lustre: lustre-MDD0000: changelog on [ 3991.094128] Lustre: lustre-MDD0001: changelog on [ 4836.045720] Lustre: lustre-MDD0001: changelog off [ 4836.727191] Lustre: 130746:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4837.075371] Lustre: lustre-MDD0000: changelog off [ 4838.880269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4838.884267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.890450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4838.893295] Lustre: Skipped 3 previous similar messages [ 4839.396372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.397150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.513434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4844.519243] Lustre: Skipped 3 previous similar messages [ 4849.634226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4849.636962] Lustre: Skipped 1 previous similar message [ 4853.215102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4853.277969] Lustre: server umount lustre-OST0000 complete [ 4854.752518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.753122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4854.756499] Lustre: Skipped 1 previous similar message [ 4860.875674] Lustre: server umount lustre-MDT0000 complete [ 4862.238419] LustreError: 130730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891568 with bad export cookie 18032535034558364422 [ 4862.241289] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4862.243398] LustreError: 130730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4862.371292] Lustre: server umount lustre-MDT0001 complete [ 4866.283587] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4867.440327] Key type lgssc unregistered [ 4867.588524] LNet: 141191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4867.592317] LNetError: 141191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4867.603364] LNet: Removed LNI 192.168.204.113@tcp [ 4867.939155] Key type .llcrypt unregistered [ 4867.940688] Key type ._llcrypt unregistered [ 4875.274270] Key type ._llcrypt registered [ 4875.275731] Key type .llcrypt registered [ 4875.319293] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4882.162838] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4882.667372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4882.690533] alg: No test for adler32 (adler32-zlib) [ 4883.576676] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 4883.686380] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4885.279129] Key type lgssc registered [ 4885.720686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4899.027308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4899.036149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.135148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4900.151668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4900.192368] Lustre: lustre-MDT0000: new disk, initializing [ 4900.236127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.246590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4901.782984] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4907.068985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.128426] Lustre: 144332:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4907.153930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4907.156454] Lustre: Skipped 1 previous similar message [ 4907.195133] Lustre: lustre-MDT0001: new disk, initializing [ 4907.242422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4907.263863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4907.269171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4909.178937] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4913.202574] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4914.870579] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4917.836487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4917.978586] Lustre: lustre-OST0000: new disk, initializing [ 4917.981147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4918.025134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.469449] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4924.152337] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4924.411985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4924.416486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4924.458172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4931.640939] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_module obdecho/obdecho [ 4932.356639] LustreError: 146945:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4932.361218] LustreError: 146945:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 4933.003383] LustreError: 147042:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4938.207925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4938.211825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.217457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4939.746321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4939.746413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.748330] Lustre: Skipped 1 previous similar message [ 4939.751158] Lustre: Skipped 1 previous similar message [ 4942.757101] Lustre: server umount lustre-MDT0000 complete [ 4944.098432] LustreError: 146071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891650 with bad export cookie 12452841909676293462 [ 4944.102253] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.103762] LustreError: 146071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4944.242431] Lustre: server umount lustre-MDT0001 complete [ 4955.945236] Lustre: server umount lustre-OST0000 complete [ 4963.861567] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4967.719050] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4981.109271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4981.211396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4981.232879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4981.273920] Lustre: lustre-MDT0000: new disk, initializing [ 4981.314736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.323967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4982.708512] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4987.324816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.393455] Lustre: 150734:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4987.427461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4987.430191] Lustre: Skipped 1 previous similar message [ 4987.473562] Lustre: lustre-MDT0001: new disk, initializing [ 4987.517418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4987.534673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4987.539059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4989.265902] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4992.940575] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4994.512889] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4997.747101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4997.865529] Lustre: lustre-OST0000: new disk, initializing [ 4997.868626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4997.905377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.273288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4998.278456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4998.330757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5000.577466] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5004.344246] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5006.170309] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5006.271825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5008.024154] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5008.109533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5013.985748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5013.988247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.990703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5013.994770] Lustre: Skipped 1 previous similar message [ 5019.106120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5019.109074] Lustre: Skipped 2 previous similar messages [ 5023.199170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5023.265751] Lustre: server umount lustre-OST0000 complete [ 5028.832203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5028.835140] LustreError: Skipped 1 previous similar message [ 5028.837062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.842769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5039.071135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5039.138226] Lustre: server umount lustre-MDT0000 complete [ 5040.676775] LustreError: 150726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891746 with bad export cookie 12452841909676294421 [ 5040.679262] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5040.682040] LustreError: 150726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5040.816123] Lustre: server umount lustre-MDT0001 complete [ 5042.193947] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 13:29:07 (1774891747) [ 5042.777322] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5043.484137] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 13:29:09 (1774891749) [ 5046.651960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.819186] LustreError: 154736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5046.828701] LustreError: 154736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5046.855975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.241605] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5051.328082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.848021] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5054.803416] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5056.118080] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5058.745878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5060.926832] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5062.911346] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5076.960157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5076.966685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.972336] Lustre: Skipped 1 previous similar message [ 5076.974952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5076.977623] Lustre: Skipped 4 previous similar messages [ 5081.696126] Lustre: server umount lustre-MDT0000 complete [ 5082.593590] LustreError: 155447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.599859] LustreError: 155447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5083.001291] LustreError: 154718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891789 with bad export cookie 12452841909676295289 [ 5083.004116] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5083.004972] LustreError: 154718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5083.611956] LustreError: 154730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.711830] LustreError: 154730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.712028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.723232] Lustre: Skipped 2 previous similar messages [ 5089.220472] Lustre: server umount lustre-MDT0001 complete [ 5091.384908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5095.851769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.943564] LustreError: 158260:0:(llog.c:1643:llog_backup()) MGC192.168.204.113@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5095.947071] Lustre: 158260:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.113@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5113.311965] LustreError: 158267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.319303] LustreError: 158267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5123.551917] LustreError: 158266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.558581] LustreError: 158266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5139.931696] LustreError: 158267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.939048] LustreError: 158267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5156.959212] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5156.962059] Lustre: 158260:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5173.727896] LustreError: 158266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5173.734106] LustreError: 158266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 5183.967305] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5183.970652] Lustre: 154728:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5197.798779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5197.802066] Lustre: Skipped 2 previous similar messages [ 5199.087438] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5201.955134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.914965] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5203.039170] LustreError: 158985:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5203.043093] LustreError: 158985:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@000000005a342f0e x1861108722928512/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1774891909 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5203.051141] LustreError: 158985:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5203.935758] LustreError: 142556:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000809e9f47 x1861108722932864/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5205.330324] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5207.151423] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5207.718585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5208.942768] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5209.064101] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5210.615788] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 5215.801300] Lustre: server umount lustre-MDT0000 complete [ 5217.160374] LustreError: 154718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774891923 with bad export cookie 12452841909676301743 [ 5217.164310] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.165274] LustreError: 154718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5217.282727] Lustre: server umount lustre-MDT0001 complete [ 5238.035555] Lustre: server umount lustre-OST0000 complete [ 5244.897721] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5248.493460] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5260.200974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.291304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5260.302165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5260.339082] Lustre: lustre-MDT0000: new disk, initializing [ 5260.368527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5261.735744] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5265.925546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.970982] Lustre: 163080:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5266.056698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5266.059992] Lustre: Skipped 2 previous similar messages [ 5266.073223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5267.448558] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5270.538681] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5271.784564] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5273.999540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.990640] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5279.062954] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5279.226618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5279.265674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5280.435655] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5280.507304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5281.725912] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5281.795453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5284.320467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5284.322215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.323206] LustreError: Skipped 1 previous similar message [ 5284.327682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5284.330556] Lustre: Skipped 9 previous similar messages [ 5288.528246] Lustre: server umount lustre-OST0000 complete [ 5304.287125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5304.353514] Lustre: server umount lustre-MDT0000 complete [ 5304.800187] LustreError: 163091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.807555] LustreError: 163091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 5305.593920] LustreError: 163857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892011 with bad export cookie 12452841909676318543 [ 5305.596651] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5305.598038] LustreError: 163857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5305.707589] Lustre: server umount lustre-MDT0001 complete [ 5308.570391] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 13:33:34 (1774892014) [ 5310.786769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5312.333788] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5315.201326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.650803] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5318.348755] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5319.624827] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5322.095268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.253053] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5326.251993] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5341.076549] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:34:06 (1774892046) [ 5345.761736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.762488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5345.763715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5345.763722] Lustre: Skipped 5 previous similar messages [ 5345.767500] Lustre: Skipped 4 previous similar messages [ 5345.773145] LustreError: Skipped 1 previous similar message [ 5348.171425] Lustre: server umount lustre-OST0000 complete [ 5355.543103] Lustre: server umount lustre-MDT0000 complete [ 5356.813938] LustreError: 167062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892062 with bad export cookie 12452841909676319411 [ 5356.816325] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.817522] LustreError: 167062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5356.923896] Lustre: server umount lustre-MDT0001 complete [ 5360.513245] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5361.528233] Key type lgssc unregistered [ 5361.646319] LNet: 170476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5361.649226] LNetError: 170476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5361.656330] LNet: Removed LNI 192.168.204.113@tcp [ 5361.957714] Key type .llcrypt unregistered [ 5361.959165] Key type ._llcrypt unregistered [ 5369.411488] Key type ._llcrypt registered [ 5369.412695] Key type .llcrypt registered [ 5369.462581] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5369.779555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5369.795066] alg: No test for adler32 (adler32-zlib) [ 5370.643271] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5370.732080] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5372.311151] Key type lgssc registered [ 5372.661102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5375.051311] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5375.056700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5390.495262] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5395.615172] LustreError: 171678:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 5400.735391] LustreError: 171678:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5400.740768] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5400.743906] LustreError: MGC192.168.204.113@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5400.748817] LustreError: 171678:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5400.752287] LustreError: 171678:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 5400.755793] LustreError: 171678:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 5400.778240] Lustre: server umount lustre-OST0000 complete [ 5400.780522] LustreError: 171678:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5403.710507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.854294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.090956] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5409.026341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.155079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5410.468572] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5412.384949] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5413.605821] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.902997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.980197] LustreError: 173767:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5415.986176] Lustre: 173767:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.204.113@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5416.031991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5417.975433] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5419.836260] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5424.610118] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.610395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5424.610823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.614171] Lustre: Skipped 1 previous similar message [ 5426.708864] Lustre: server umount lustre-MDT0000 complete [ 5428.042359] LustreError: 172159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892134 with bad export cookie 13495585404459403247 [ 5428.045690] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5428.046949] LustreError: 172159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5428.155581] Lustre: server umount lustre-MDT0001 complete [ 5439.700947] Lustre: server umount lustre-OST0000 complete [ 5442.641701] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5445.955499] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5458.691522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.779467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5458.791579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5458.828477] Lustre: lustre-MDT0000: new disk, initializing [ 5458.851583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.858459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5460.084852] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5464.349312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.387202] Lustre: 177143:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5464.409285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5464.411534] Lustre: Skipped 1 previous similar message [ 5464.445942] Lustre: lustre-MDT0001: new disk, initializing [ 5464.475404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5464.486066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5464.489214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5465.908512] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5469.040750] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5470.337780] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5472.640353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5472.733959] Lustre: lustre-OST0000: new disk, initializing [ 5472.737942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5472.765237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5474.594222] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5474.872114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5474.875809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5474.917719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5477.808557] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5479.295078] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5479.387889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5480.941078] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5481.017645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5485.025436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.025914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5485.029491] Lustre: Skipped 1 previous similar message [ 5485.033298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5485.036878] Lustre: Skipped 3 previous similar messages [ 5487.758468] Lustre: server umount lustre-OST0000 complete [ 5490.144733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5490.144804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.151437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.153693] Lustre: Skipped 1 previous similar message [ 5495.122928] Lustre: server umount lustre-MDT0000 complete [ 5495.263891] LustreError: 177153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.268367] LustreError: 177153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5496.321263] LustreError: 177135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892202 with bad export cookie 13495585404459403947 [ 5496.324597] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5496.325338] LustreError: 177135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5496.431972] Lustre: server umount lustre-MDT0001 complete [ 5499.269957] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 13:36:45 (1774892205) [ 5504.948492] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5505.935981] Key type lgssc unregistered [ 5506.059339] LNet: 181657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5506.062370] LNetError: 181657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5506.070249] LNet: Removed LNI 192.168.204.113@tcp [ 5506.351085] Key type .llcrypt unregistered [ 5506.352420] Key type ._llcrypt unregistered [ 5512.977214] Key type ._llcrypt registered [ 5512.978261] Key type .llcrypt registered [ 5513.010867] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5513.385180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5513.390172] alg: No test for adler32 (adler32-zlib) [ 5514.236068] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5514.309636] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5515.879151] Key type lgssc registered [ 5516.207911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5518.658073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5518.663631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.800993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5520.978568] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5523.705648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.815333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5525.007990] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5526.672513] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5527.792377] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5530.204884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5530.272163] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5530.274353] LustreError: 184484:0:(llog.c:1684:llog_backup()) MGC192.168.204.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5530.278925] Lustre: 184484:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.204.113@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5530.315231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.208584] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5533.876605] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5538.785176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5538.786713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5538.789508] Lustre: Skipped 2 previous similar messages [ 5538.792872] Lustre: Skipped 2 previous similar messages [ 5540.688980] Lustre: server umount lustre-MDT0000 complete [ 5541.792350] LustreError: 185131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892247 with bad export cookie 5304575124560985646 [ 5541.795371] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5541.797087] LustreError: 185131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5541.905157] Lustre: server umount lustre-MDT0001 complete [ 5553.365589] Lustre: server umount lustre-OST0000 complete [ 5556.159448] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5559.084368] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5570.607847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5570.693705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5570.705503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5570.742279] Lustre: lustre-MDT0000: new disk, initializing [ 5570.765324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.771441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5572.048233] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5576.011867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.053433] Lustre: 187860:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5576.071643] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5576.072954] Lustre: Skipped 1 previous similar message [ 5576.074316] LustreError: 188624:0:(llog.c:1684:llog_backup()) MGC192.168.204.113@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5576.077807] LustreError: 188624:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5576.080151] Lustre: 188624:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.113@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5576.088461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5576.090526] Lustre: Skipped 1 previous similar message [ 5576.121970] Lustre: lustre-MDT0001: new disk, initializing [ 5576.153455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5576.165209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5576.168376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5577.372354] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5580.452402] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5581.670149] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5583.851486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.938687] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5583.940300] Lustre: Skipped 1 previous similar message [ 5583.942406] LustreError: 189577:0:(llog.c:1684:llog_backup()) MGC192.168.204.113@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5583.946764] LustreError: 189577:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5583.949810] Lustre: 189577:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.113@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5583.954276] Lustre: 189577:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5583.963371] Lustre: lustre-OST0000: new disk, initializing [ 5583.966428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5585.811196] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5588.874046] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5590.002762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5590.005968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5590.034885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5590.205335] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5590.273857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5591.572515] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5591.663101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5595.106432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5595.106495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.109126] LustreError: Skipped 1 previous similar message [ 5595.109387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.112330] Lustre: Skipped 1 previous similar message [ 5598.413244] Lustre: server umount lustre-OST0000 complete [ 5600.224570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.225104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.228122] Lustre: Skipped 1 previous similar message [ 5600.231173] Lustre: Skipped 2 previous similar messages [ 5605.343955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5605.346452] Lustre: Skipped 1 previous similar message [ 5605.719719] Lustre: server umount lustre-MDT0000 complete [ 5606.939105] LustreError: 187852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892313 with bad export cookie 5304575124560986332 [ 5606.942249] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.943822] LustreError: 187852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5607.055039] Lustre: server umount lustre-MDT0001 complete [ 5609.811840] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 13:38:35 (1774892315) [ 5614.768943] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5618.168976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.299312] LustreError: 192559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.307089] LustreError: 192559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5618.314739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.317802] Lustre: Skipped 1 previous similar message [ 5619.631154] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5622.672291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.149921] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5625.117917] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5627.366533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5629.368654] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5632.130634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5632.191835] Lustre: lustre-OST0001: new disk, initializing [ 5632.193644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5633.595967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5633.599029] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5633.642055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5634.116201] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5639.513584] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5643.103319] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5648.674692] Lustre: DEBUG MARKER: ADD OST3 [ 5650.335580] loop: module loaded [ 5650.353797] loop0: detected capacity change from 0 to 204800000 [ 5650.424477] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5650.429640] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5650.433509] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5652.593846] loop0: detected capacity change from 0 to 204800000 [ 5654.511269] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5654.572897] Lustre: lustre-OST0003: new disk, initializing [ 5654.575177] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5654.611410] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5654.614078] Lustre: Skipped 3 previous similar messages [ 5655.392959] Lustre: DEBUG MARKER: STOP OST3 [ 5655.884644] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5655.919092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5655.922643] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5655.932819] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5656.034078] Lustre: Failing over lustre-OST0003 [ 5656.071721] Lustre: server umount lustre-OST0003 complete [ 5656.652389] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5657.887154] LustreError: 196835:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5660.461533] Lustre: DEBUG MARKER: START OST3 again [ 5661.158102] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5661.158504] LustreError: 194004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.160351] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5661.162203] Lustre: Skipped 1 previous similar message [ 5661.167885] LustreError: 194004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5662.631491] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5662.717829] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5663.850910] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5664.364190] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5664.364481] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5664.481845] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5677.771020] Lustre: Failing over lustre-OST0003 [ 5677.804519] Lustre: server umount lustre-OST0003 complete [ 5678.058993] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.060148] LustreError: 194006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.063493] Lustre: Skipped 1 previous similar message [ 5683.172229] LustreError: 194004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.177140] LustreError: 194004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5688.292269] LustreError: 194006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.297713] LustreError: 194006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5693.410797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.412735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5693.414598] Lustre: Skipped 2 previous similar messages [ 5693.418521] Lustre: Skipped 3 previous similar messages [ 5696.346256] Lustre: server umount lustre-MDT0000 complete [ 5697.522370] LustreError: 193274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892403 with bad export cookie 5304575124560987200 [ 5697.525573] LustreError: 193274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5697.525968] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.528661] LustreError: 194024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5698.532724] LustreError: 194024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5702.560815] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5702.563298] Lustre: Skipped 3 previous similar messages [ 5703.806239] Lustre: server umount lustre-MDT0001 complete [ 5705.227511] Lustre: server umount lustre-OST0000 complete [ 5706.565400] Lustre: server umount lustre-OST0001 complete [ 5708.699803] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5711.156137] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5722.517062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.621560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5722.633294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5722.669637] Lustre: lustre-MDT0000: new disk, initializing [ 5722.692532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.695228] Lustre: Skipped 1 previous similar message [ 5722.702153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5723.901357] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5727.880768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5727.921373] Lustre: 201880:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5728.034941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5729.229891] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5732.176235] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5733.279945] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5735.348523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5737.017280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5737.050598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5737.193893] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5740.190111] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5741.456397] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5741.519521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5742.763258] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5742.828255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5747.168602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5747.169315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.171425] LustreError: Skipped 1 previous similar message [ 5747.176015] Lustre: Skipped 3 previous similar messages [ 5747.176575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5747.181080] Lustre: Skipped 1 previous similar message [ 5749.515792] Lustre: server umount lustre-OST0000 complete [ 5756.821392] Lustre: server umount lustre-MDT0000 complete [ 5757.408281] LustreError: 201890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.412464] LustreError: 201890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5758.034623] LustreError: 201872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892464 with bad export cookie 5304575124560989454 [ 5758.037697] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5758.039680] LustreError: 201872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5758.157433] Lustre: server umount lustre-MDT0001 complete [ 5759.279100] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:41:05 (1774892465) [ 5764.070929] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5766.933231] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5778.210513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.301182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5778.313058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5778.314789] Lustre: Skipped 3 previous similar messages [ 5778.350787] Lustre: lustre-MDT0000: new disk, initializing [ 5778.353012] Lustre: Skipped 2 previous similar messages [ 5778.387197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5778.391105] Lustre: Skipped 2 previous similar messages [ 5779.643023] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5783.871931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.915214] Lustre: 208040:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5784.007937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5785.224323] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5788.294199] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5789.439372] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5791.640247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5791.765547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.768295] Lustre: Skipped 4 previous similar messages [ 5793.050129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5793.538692] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5796.533412] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5797.803444] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5797.878705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5799.084205] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5799.148913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5803.491890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5803.492763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5803.493579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5803.493584] Lustre: Skipped 2 previous similar messages [ 5803.496545] Lustre: Skipped 4 previous similar messages [ 5814.239084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5814.290323] Lustre: server umount lustre-OST0000 complete [ 5821.650161] Lustre: server umount lustre-MDT0000 complete [ 5822.831334] LustreError: 208031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892528 with bad export cookie 5304575124560990336 [ 5822.834386] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5822.835220] LustreError: 208031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5822.956875] Lustre: server umount lustre-MDT0001 complete [ 5825.606572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.741776] LustreError: 211852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.936791] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5829.476113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.758922] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5832.482770] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5833.610561] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5835.740436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5837.598881] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5839.378960] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5859.807850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.810555] LustreError: Skipped 1 previous similar message [ 5862.157894] Lustre: server umount lustre-OST0000 complete [ 5869.459489] Lustre: server umount lustre-MDT0000 complete [ 5870.663075] LustreError: 211833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892576 with bad export cookie 5304575124560991218 [ 5870.665823] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5870.667161] LustreError: 211833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5870.775692] Lustre: server umount lustre-MDT0001 complete [ 5971.688269] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5972.677553] Key type lgssc unregistered [ 5972.793340] LNet: 215181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5972.795465] LNetError: 215181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5972.804320] LNet: Removed LNI 192.168.204.113@tcp [ 5973.105100] Key type .llcrypt unregistered [ 5973.106689] Key type ._llcrypt unregistered [ 5980.785505] Key type ._llcrypt registered [ 5980.786821] Key type .llcrypt registered [ 5980.821912] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5986.788356] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5987.068705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5987.090485] alg: No test for adler32 (adler32-zlib) [ 5987.928639] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 5988.000663] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5989.575137] Key type lgssc registered [ 5989.906463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6000.460828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6000.468833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.565103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6001.576904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6001.612764] Lustre: lustre-MDT0000: new disk, initializing [ 6001.636947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.644422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6002.983204] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6007.143307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.192091] Lustre: 218471:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6007.215937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6007.218108] Lustre: Skipped 1 previous similar message [ 6007.262218] Lustre: lustre-MDT0001: new disk, initializing [ 6007.294822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.307903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6007.311855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6008.541362] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6011.642788] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6012.969256] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6015.252298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.364797] Lustre: lustre-OST0000: new disk, initializing [ 6015.366806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6015.396807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.376663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6016.380296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6016.427179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6017.257834] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6020.298223] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6021.709321] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6021.783892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6023.144277] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6023.221231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6026.722537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6026.723138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.723834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6026.726231] LustreError: Skipped 1 previous similar message [ 6026.736523] Lustre: Skipped 1 previous similar message [ 6029.965474] Lustre: server umount lustre-OST0000 complete [ 6031.840413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.841206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.845786] Lustre: Skipped 1 previous similar message [ 6031.847852] Lustre: Skipped 1 previous similar message [ 6036.960129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6036.963135] Lustre: Skipped 2 previous similar messages [ 6037.396748] Lustre: server umount lustre-MDT0000 complete [ 6038.665589] LustreError: 218465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892744 with bad export cookie 4658206596454398598 [ 6038.668056] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6038.669072] LustreError: 218465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6038.807586] Lustre: server umount lustre-MDT0001 complete [ 6044.922347] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6048.337297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.487406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.754429] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6052.719282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.826650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6054.168439] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6055.105350] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6057.279587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6059.183675] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6061.900138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6061.959398] Lustre: lustre-OST0001: new disk, initializing [ 6061.962602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6061.990611] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6061.992559] Lustre: Skipped 1 previous similar message [ 6063.677202] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6063.679514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6063.683029] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6063.712689] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6068.959429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6074.533872] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6083.208221] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:46:28 (1774892788) [ 6084.065237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.068504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.069408] Lustre: Skipped 3 previous similar messages [ 6084.071949] Lustre: Skipped 2 previous similar messages [ 6089.184374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6089.187162] Lustre: Skipped 2 previous similar messages [ 6090.004088] Lustre: server umount lustre-MDT0000 complete [ 6091.217305] LustreError: 225986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892797 with bad export cookie 4658206596454399410 [ 6091.220950] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.222247] LustreError: 225986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6091.336052] Lustre: server umount lustre-MDT0001 complete [ 6102.529533] Lustre: server umount lustre-OST0000 complete [ 6113.952197] Lustre: server umount lustre-OST0001 complete [ 6116.016241] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6119.425666] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6131.304981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.402911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6131.414917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6131.458262] Lustre: lustre-MDT0000: new disk, initializing [ 6131.485357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6131.493852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6132.832974] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6136.993655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.039749] Lustre: 229121:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6137.141607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6138.437552] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6141.598760] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6142.811216] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6144.893294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6144.986759] Lustre: lustre-OST0000: new disk, initializing [ 6144.988482] Lustre: Skipped 1 previous similar message [ 6144.990503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6144.992814] Lustre: Skipped 2 previous similar messages [ 6146.730908] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6146.867509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6146.869764] Lustre: Skipped 1 previous similar message [ 6146.871333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6146.905101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6149.793451] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6151.211085] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6151.282035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6152.720923] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6152.802557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6157.281319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6157.281986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.284419] LustreError: Skipped 1 previous similar message [ 6157.285036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.289384] Lustre: Skipped 1 previous similar message [ 6157.296346] Lustre: Skipped 1 previous similar message [ 6159.568654] Lustre: server umount lustre-OST0000 complete [ 6162.400783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.406451] Lustre: Skipped 1 previous similar message [ 6175.199089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6175.264170] Lustre: server umount lustre-MDT0000 complete [ 6176.473434] LustreError: 229113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774892882 with bad export cookie 4658206596454401510 [ 6176.477593] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6176.479549] LustreError: 229113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6176.590463] Lustre: server umount lustre-MDT0001 complete [ 6179.386043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6179.545658] LustreError: 232929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.552267] LustreError: 232929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6179.571922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.573940] Lustre: Skipped 2 previous similar messages [ 6180.869118] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6183.561665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.816781] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6186.390904] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6187.492417] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6189.597856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6191.398066] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6193.029104] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6337.505322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6337.506571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.508133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.511708] Lustre: Skipped 1 previous similar message [ 6337.515555] Lustre: Skipped 7 previous similar messages [ 6343.432874] Lustre: server umount lustre-OST0000 complete [ 6347.744566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6350.743224] Lustre: server umount lustre-MDT0000 complete [ 6351.977104] LustreError: 232955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893058 with bad export cookie 4658206596454402392 [ 6351.980628] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6351.984870] LustreError: 232955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6352.095269] Lustre: server umount lustre-MDT0001 complete [ 6458.269579] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6459.284235] Key type lgssc unregistered [ 6459.414387] LNet: 236191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.417889] LNetError: 236191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.430322] LNet: Removed LNI 192.168.204.113@tcp [ 6459.755116] Key type .llcrypt unregistered [ 6459.757300] Key type ._llcrypt unregistered [ 6467.891432] Key type ._llcrypt registered [ 6467.892751] Key type .llcrypt registered [ 6467.933113] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6474.401193] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6474.751230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6474.780071] alg: No test for adler32 (adler32-zlib) [ 6475.651993] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 6475.742390] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6477.319423] Key type lgssc registered [ 6477.702261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6491.755766] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6495.133721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6495.140542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.232393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6496.245142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6496.280397] Lustre: lustre-MDT0000: new disk, initializing [ 6496.301160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6496.307721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6497.528936] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6501.497818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.539798] Lustre: 240180:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6501.561511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6501.563150] Lustre: Skipped 1 previous similar message [ 6501.593510] Lustre: lustre-MDT0001: new disk, initializing [ 6501.625029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6501.637824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6501.640704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6502.843995] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6504.990820] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6507.067260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6507.154659] Lustre: lustre-OST0000: new disk, initializing [ 6507.157312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6507.183543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.842878] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6511.104367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6511.109034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6511.155542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6513.014636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.077903] Lustre: lustre-OST0001: new disk, initializing [ 6513.079896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6513.109554] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6514.918871] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6518.268856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6518.271885] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6518.297746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6520.276748] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6526.026271] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6537.695896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6537.698027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6537.701072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.723084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.723442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.727428] Lustre: Skipped 1 previous similar message [ 6538.730728] Lustre: Skipped 3 previous similar messages [ 6540.009343] Lustre: server umount lustre-MDT0000 complete [ 6541.223228] LustreError: 240172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893247 with bad export cookie 16315859539545221963 [ 6541.226122] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6541.226872] LustreError: 240172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6541.360512] Lustre: server umount lustre-MDT0001 complete [ 6552.004208] Lustre: server umount lustre-OST0000 complete [ 6563.491699] Lustre: server umount lustre-OST0001 complete [ 6568.265474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6571.614639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6574.526466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6577.686504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6584.012487] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6587.505061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.513579] Lustre: lustre-MDT0000: reset Object Index mappings [ 6587.676830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.997382] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6591.984924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.991684] Lustre: lustre-MDT0001: reset Object Index mappings [ 6593.525493] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6594.578331] Lustre: 248712:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6596.891193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6597.012112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6597.014889] Lustre: Skipped 1 previous similar message [ 6598.052351] LustreError: 249254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6598.055516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6598.059322] LustreError: 249254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6598.946735] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6601.901531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.052603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6603.874421] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6607.044576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6608.215372] Lustre: 250554:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6618.079886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6618.081927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.085994] Lustre: Skipped 1 previous similar message [ 6618.088272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6620.445575] Lustre: server umount lustre-MDT0000 complete [ 6621.622384] LustreError: 247586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893327 with bad export cookie 16315859539545228739 [ 6621.625844] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6621.627580] LustreError: 247586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6621.740553] Lustre: server umount lustre-MDT0001 complete [ 6633.923613] Lustre: server umount lustre-OST0000 complete [ 6645.464518] Lustre: server umount lustre-OST0001 complete [ 6651.783750] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:55:57 (1774893357) [ 6657.005665] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6660.339371] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6672.187754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.272021] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6672.282784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6672.315204] Lustre: lustre-MDT0000: new disk, initializing [ 6672.335372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.337211] Lustre: Skipped 1 previous similar message [ 6672.341902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6673.521418] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6677.489667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.524325] Lustre: 254768:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6677.622454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6678.813885] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6681.773859] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6682.904382] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6685.074920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6685.167589] Lustre: lustre-OST0000: new disk, initializing [ 6685.169186] Lustre: Skipped 1 previous similar message [ 6685.170860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6685.173074] Lustre: Skipped 2 previous similar messages [ 6686.391862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6686.394166] Lustre: Skipped 1 previous similar message [ 6686.395780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6686.428162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6686.938159] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6689.784742] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6690.982831] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6691.039550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6692.264499] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6692.332663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6696.930029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6696.930078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.931295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.931299] Lustre: Skipped 3 previous similar messages [ 6696.933805] LustreError: Skipped 1 previous similar message [ 6696.937200] Lustre: Skipped 4 previous similar messages [ 6699.027553] Lustre: server umount lustre-OST0000 complete [ 6702.049073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6702.050312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6702.052465] Lustre: Skipped 1 previous similar message [ 6702.056788] Lustre: Skipped 2 previous similar messages [ 6706.396485] Lustre: server umount lustre-MDT0000 complete [ 6707.167873] LustreError: 254779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6707.172195] LustreError: 254779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6707.580161] LustreError: 254758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893413 with bad export cookie 16315859539545230832 [ 6707.583412] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6707.586053] LustreError: 254758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6707.692326] Lustre: server umount lustre-MDT0001 complete [ 6713.447579] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6716.756294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.911323] LustreError: 259272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.925783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.927945] Lustre: Skipped 2 previous similar messages [ 6718.195425] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6720.910467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.202102] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6723.111746] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6725.311513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.232200] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6730.035971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6730.108670] Lustre: lustre-OST0001: new disk, initializing [ 6730.110768] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6731.207044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6731.210043] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6731.238445] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6732.021240] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6736.410248] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6737.548335] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6744.188201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6744.199923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6746.594169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.596640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6746.597654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6746.599789] Lustre: Skipped 2 previous similar messages [ 6752.019724] Lustre: server umount lustre-MDT0000 complete [ 6753.101330] LustreError: 259253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893459 with bad export cookie 16315859539545231714 [ 6753.106542] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.226553] Lustre: server umount lustre-MDT0001 complete [ 6764.042036] Lustre: server umount lustre-OST0000 complete [ 6775.521629] Lustre: server umount lustre-OST0001 complete [ 6780.382755] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6783.481936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.601916] LustreError: 264547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6783.619722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.622229] Lustre: Skipped 3 previous similar messages [ 6784.873614] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6787.545566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.931871] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6789.818169] Lustre: 265654:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6791.934047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6793.123117] LustreError: 266010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.129264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6793.926611] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6796.846065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6798.012814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6798.948084] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6802.118103] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6803.286670] Lustre: 267504:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6809.978466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6809.989332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6812.975843] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:58:38 (1774893518) [ 6828.511871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6828.514848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.519404] Lustre: Skipped 1 previous similar message [ 6828.521509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.523697] Lustre: Skipped 7 previous similar messages [ 6830.614804] Lustre: server umount lustre-MDT0000 complete [ 6831.703052] LustreError: 265273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893537 with bad export cookie 16315859539545238028 [ 6831.706277] LustreError: 265273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6831.706464] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6834.143860] LustreError: 266030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6834.148797] LustreError: 266030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6837.866209] Lustre: server umount lustre-MDT0001 complete [ 6845.244263] Lustre: server umount lustre-OST0000 complete [ 6852.705924] Lustre: server umount lustre-OST0001 complete [ 6854.676186] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6857.754944] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6871.908079] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6875.509321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.603926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6875.617784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6875.656643] Lustre: lustre-MDT0000: new disk, initializing [ 6875.686482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6876.984410] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6881.163893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6881.209255] Lustre: 271684:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6881.331247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6882.674243] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6887.071674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.406139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6889.160493] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6893.239683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6894.635583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6895.220955] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6899.773453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6910.305678] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 14:00:16 (1774893616) [ 6911.967904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6911.970294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.973986] Lustre: Skipped 5 previous similar messages [ 6911.975706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6911.977417] Lustre: Skipped 8 previous similar messages [ 6917.215390] Lustre: server umount lustre-MDT0000 complete [ 6918.472420] LustreError: 271676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893624 with bad export cookie 16315859539545245910 [ 6918.474637] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6918.478983] LustreError: 271676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6918.624338] Lustre: server umount lustre-MDT0001 complete [ 6930.377237] Lustre: server umount lustre-OST0000 complete [ 6941.914443] Lustre: server umount lustre-OST0001 complete [ 6948.033545] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6951.306239] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6963.747907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.867454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6963.870271] Lustre: Skipped 2 previous similar messages [ 6963.880495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6963.882625] Lustre: Skipped 4 previous similar messages [ 6963.922609] Lustre: lustre-MDT0000: new disk, initializing [ 6963.924224] Lustre: Skipped 3 previous similar messages [ 6963.954589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.958434] Lustre: Skipped 7 previous similar messages [ 6963.966793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6963.971300] Lustre: Skipped 3 previous similar messages [ 6965.354073] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6969.600382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.644042] Lustre: 278732:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6969.745285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6969.748029] Lustre: Skipped 2 previous similar messages [ 6971.149076] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6974.282381] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6975.514412] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6977.742969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6979.297447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6979.855160] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6982.807549] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6983.545698] Lustre: Mounted lustre-client [ 6983.861793] LustreError: 281226:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9590fa1d2000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6983.888969] Lustre: Unmounted lustre-client [ 6988.770868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.771943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6988.775115] Lustre: Skipped 4 previous similar messages [ 6988.780262] Lustre: Skipped 5 previous similar messages [ 6988.948494] Lustre: server umount lustre-OST0000 complete [ 6990.303962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6996.383904] Lustre: server umount lustre-MDT0000 complete [ 6997.704399] LustreError: 278722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893703 with bad export cookie 16315859539545247436 [ 6997.708727] LustreError: 278722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6997.708739] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.847626] Lustre: server umount lustre-MDT0001 complete [ 7001.767334] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7002.787677] Key type lgssc unregistered [ 7041.823346] LNet: 282504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7041.826077] LNetError: 282504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7116.581501] LNet: Removed LNI 192.168.204.113@tcp [ 7116.912121] Key type .llcrypt unregistered [ 7116.913109] Key type ._llcrypt unregistered [ 7122.915125] Key type ._llcrypt registered [ 7122.917210] Key type .llcrypt registered [ 7122.961376] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 7129.731141] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7130.095768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7130.146449] alg: No test for adler32 (adler32-zlib) [ 7130.994686] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7131.081627] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7132.663094] Key type lgssc registered [ 7133.031493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7143.189197] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:04:08 (1774893848) [ 7145.485357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7145.490267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.575854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7146.585919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7146.621266] Lustre: lustre-MDT0000: new disk, initializing [ 7146.642839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.648927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7147.853463] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7151.953318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.993520] Lustre: 285666:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7152.017354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7152.019679] Lustre: Skipped 1 previous similar message [ 7152.054753] Lustre: lustre-MDT0001: new disk, initializing [ 7152.085565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.097117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7152.099996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7153.374864] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7156.445102] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7157.576395] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7159.872570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.994129] Lustre: lustre-OST0000: new disk, initializing [ 7159.996822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7160.039196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.116808] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7165.381827] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7165.945217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7165.949758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7165.980149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7176.162882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.163218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.163870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.167181] LustreError: Skipped 1 previous similar message [ 7176.171641] Lustre: Skipped 1 previous similar message [ 7181.070464] Lustre: server umount lustre-OST0000 complete [ 7182.815993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7182.818643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7182.824129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7182.825824] Lustre: Skipped 1 previous similar message [ 7186.400439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.401293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.513089] Lustre: server umount lustre-MDT0000 complete [ 7189.801201] LustreError: 286443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893895 with bad export cookie 12171862098757405962 [ 7189.804692] LustreError: 286443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7189.805303] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7189.936981] Lustre: server umount lustre-MDT0001 complete [ 7193.395424] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7194.439968] Key type lgssc unregistered [ 7194.575499] LNet: 289039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7194.578144] LNetError: 289039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7194.588491] LNet: Removed LNI 192.168.204.113@tcp [ 7194.920324] Key type .llcrypt unregistered [ 7194.921946] Key type ._llcrypt unregistered [ 7204.928580] Key type ._llcrypt registered [ 7204.930014] Key type .llcrypt registered [ 7204.967302] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7205.315112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7205.354466] alg: No test for adler32 (adler32-zlib) [ 7206.210812] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7206.295832] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7207.887163] Key type lgssc registered [ 7208.225539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7210.618480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7210.623389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.780563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.951203] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7215.575558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.693067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7216.852567] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7218.453206] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7219.519647] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7221.642692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.755626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.428058] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7225.070774] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7233.897228] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7234.428603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7236.064836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7236.066246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.067429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.078343] Lustre: Skipped 1 previous similar message [ 7236.578568] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 7236.581985] Lustre: Skipped 2 previous similar messages [ 7241.185956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7241.187679] Lustre: Skipped 2 previous similar messages [ 7241.948906] Lustre: server umount lustre-MDT0000 complete [ 7243.159962] LustreError: 290396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774893949 with bad export cookie 13765812174042363116 [ 7243.163332] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.163501] LustreError: 290396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7246.304080] LustreError: 292562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.304301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.310440] LustreError: 292562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7246.314916] Lustre: Skipped 1 previous similar message [ 7246.315559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7246.320624] Lustre: Skipped 1 previous similar message [ 7246.812175] LustreError: 290409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7248.352019] LustreError: 292562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.336612] Lustre: server umount lustre-MDT0001 complete [ 7251.938689] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 7251.942068] Lustre: Skipped 2 previous similar messages [ 7256.791744] Lustre: server umount lustre-OST0000 complete [ 7259.697668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.026387] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7261.933198] Lustre: 294145:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.113@tcp [ 7262.696335] Lustre: server umount MGS complete [ 7265.581800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7265.723694] LustreError: 294716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7265.745048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.025922] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7269.809354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.936606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7271.248713] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7274.074392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.025081] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7277.774227] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7293.818644] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 15 sec [ 7294.994776] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7295.547039] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7298.709426] Lustre: server umount lustre-OST0000 complete [ 7300.064797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.065545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.070175] Lustre: Skipped 1 previous similar message [ 7300.075121] Lustre: Skipped 1 previous similar message [ 7310.328065] Lustre: server umount lustre-MDT0000 complete [ 7311.543448] LustreError: 296893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774894017 with bad export cookie 13765812174042363942 [ 7311.546867] LustreError: 296893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7311.547553] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7321.102354] Lustre: server umount lustre-MDT0001 complete [ 7324.496349] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7325.470630] Key type lgssc unregistered [ 7325.588279] LNet: 297916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7325.590610] LNetError: 297916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7325.598330] LNet: Removed LNI 192.168.204.113@tcp [ 7325.878288] Key type .llcrypt unregistered [ 7325.880122] Key type ._llcrypt unregistered [ 7331.634659] Key type ._llcrypt registered [ 7331.636254] Key type .llcrypt registered [ 7331.675712] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 7338.075379] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7338.411301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7338.452374] alg: No test for adler32 (adler32-zlib) [ 7339.319733] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7339.402930] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7340.975127] Key type lgssc registered [ 7341.292226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7352.441601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7352.449360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.534078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7353.544873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7353.580372] Lustre: lustre-MDT0000: new disk, initializing [ 7353.603563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.610279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7354.882328] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7358.888595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.930134] Lustre: 301012:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7358.954903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7358.957105] Lustre: Skipped 1 previous similar message [ 7358.991922] Lustre: lustre-MDT0001: new disk, initializing [ 7359.025666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7359.036454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7359.041781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7360.272618] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7363.319087] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7364.528220] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7366.873308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.981446] Lustre: lustre-OST0000: new disk, initializing [ 7366.984811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7367.015074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.014137] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7372.237527] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7372.282989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7372.287189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7372.323436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7373.621904] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7373.694351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7375.013466] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7375.087900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7377.378434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7377.378977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.382168] LustreError: Skipped 1 previous similar message [ 7377.387065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7377.390369] Lustre: Skipped 1 previous similar message [ 7381.842220] Lustre: server umount lustre-OST0000 complete [ 7384.544176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7384.546527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.549937] Lustre: Skipped 1 previous similar message [ 7384.551760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7387.616310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7387.616720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7389.282558] Lustre: server umount lustre-MDT0000 complete [ 7390.560354] LustreError: 301002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774894096 with bad export cookie 16297317871995466876 [ 7390.563729] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7390.565096] LustreError: 301002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7390.704591] Lustre: server umount lustre-MDT0001 complete [ 7393.730291] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 14:08:19 (1774894099) [ 7411.751503] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 14:08:37 (1774894117) [ 7426.635362] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7427.582411] Key type lgssc unregistered [ 7427.695310] LNet: 307548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7427.698208] LNetError: 307548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7427.707354] LNet: Removed LNI 192.168.204.113@tcp [ 7428.009441] Key type .llcrypt unregistered [ 7428.011209] Key type ._llcrypt unregistered [ 7454.558867] Key type ._llcrypt registered [ 7454.559930] Key type .llcrypt registered [ 7454.602088] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7454.705414] Key type .llcrypt unregistered [ 7454.706425] Key type ._llcrypt unregistered [ 7465.303677] Key type ._llcrypt registered [ 7465.304810] Key type .llcrypt registered [ 7465.338891] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7465.436192] Key type .llcrypt unregistered [ 7465.437165] Key type ._llcrypt unregistered [ 7476.488955] Key type ._llcrypt registered [ 7476.490383] Key type .llcrypt registered [ 7476.529694] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7476.627382] Key type .llcrypt unregistered [ 7476.629245] Key type ._llcrypt unregistered [ 7503.476017] Key type ._llcrypt registered [ 7503.477145] Key type .llcrypt registered [ 7503.518846] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7503.624289] Key type .llcrypt unregistered [ 7503.625624] Key type ._llcrypt unregistered [ 7514.900777] Key type ._llcrypt registered [ 7514.901990] Key type .llcrypt registered [ 7514.941088] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7515.056297] Key type .llcrypt unregistered [ 7515.058542] Key type ._llcrypt unregistered [ 7525.960527] Key type ._llcrypt registered [ 7525.961638] Key type .llcrypt registered [ 7526.007240] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7526.114659] Key type .llcrypt unregistered [ 7526.115589] Key type ._llcrypt unregistered [ 7537.416372] Key type ._llcrypt registered [ 7537.417326] Key type .llcrypt registered [ 7537.455195] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7537.559234] Key type .llcrypt unregistered [ 7537.560897] Key type ._llcrypt unregistered [ 7548.858138] Key type ._llcrypt registered [ 7548.859246] Key type .llcrypt registered [ 7548.896153] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7549.002229] Key type .llcrypt unregistered [ 7549.004202] Key type ._llcrypt unregistered [ 7560.745393] Key type ._llcrypt registered [ 7560.746569] Key type .llcrypt registered [ 7560.787738] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7560.889347] Key type .llcrypt unregistered [ 7560.890824] Key type ._llcrypt unregistered [ 7582.045774] Key type ._llcrypt registered [ 7582.046951] Key type .llcrypt registered [ 7582.087335] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7582.189459] Key type .llcrypt unregistered [ 7582.190699] Key type ._llcrypt unregistered [ 7593.040825] Key type ._llcrypt registered [ 7593.041927] Key type .llcrypt registered [ 7593.082119] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7593.186377] Key type .llcrypt unregistered [ 7593.188177] Key type ._llcrypt unregistered [ 7604.000144] Key type ._llcrypt registered [ 7604.001135] Key type .llcrypt registered [ 7604.039794] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7604.147474] Key type .llcrypt unregistered [ 7604.149679] Key type ._llcrypt unregistered [ 7623.353326] Key type ._llcrypt registered [ 7623.354410] Key type .llcrypt registered [ 7623.390436] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 7629.160310] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7629.579282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7629.654885] alg: No test for adler32 (adler32-zlib) [ 7630.519966] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 7630.610291] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7632.199197] Key type lgssc registered [ 7632.571592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7642.552847] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7329 sec ========== 14:12:28 (1774894348) [ 7643.084508] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:12:28 (1774894348) === [ 7644.128718] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:12:29 (1774894349) === [ 7656.915381] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7657.883569] Key type lgssc unregistered [ 7658.008360] LNet: 326353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7658.011081] LNetError: 326353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7658.019258] LNet: Removed LNI 192.168.204.113@tcp [ 7658.293329] Key type .llcrypt unregistered [ 7658.294470] Key type ._llcrypt unregistered