[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454408929 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002381] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007268] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.012044] Yama: becoming mindful. [ 0.013027] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022643] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028182] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032061] *** VALIDATE tmpfs *** [ 0.033000] *** VALIDATE proc *** [ 0.034223] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.037035] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038102] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040021] Spectre V2 : User space: Vulnerable [ 0.041004] Speculative Store Bypass: Vulnerable [ 0.044402] debug: unmapping init [mem 0xffffffffad059000-0xffffffffad060fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047411] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048014] ... version: 2 [ 0.049006] ... bit width: 48 [ 0.049946] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052008] ... fixed-purpose events: 3 [ 0.052850] ... event mask: 000000070000000f [ 0.053255] rcu: Hierarchical SRCU implementation. [ 0.055325] smp: Bringing up secondary CPUs ... [ 0.056449] x86: Booting SMP configuration: [ 0.057011] .... node #0, CPUs: #1 #2 #3 [ 0.061884] smp: Brought up 1 node, 4 CPUs [ 0.062939] smpboot: Max logical packages: 1 [ 0.063010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129017] node 0 deferred pages initialised in 64ms [ 0.133110] devtmpfs: initialized [ 0.135859] x86/mm: Memory block size: 128MB [ 0.143786] gcov: version magic: 0x41383552 [ 0.148324] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162412] pinctrl core: initialized pinctrl subsystem [ 0.164133] [ 0.164647] ************************************************************* [ 0.166010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168009] ** ** [ 0.170006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171007] ** ** [ 0.173011] ** This means that this kernel is built to expose internal ** [ 0.176012] ** IOMMU data structures, which may compromise security on ** [ 0.178011] ** your system. ** [ 0.180028] ** ** [ 0.182011] ** If you see this message and you are not debugging the ** [ 0.183000] ** kernel, report this immediately to your vendor! ** [ 0.184011] ** ** [ 0.186013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188016] ************************************************************* [ 0.189562] NET: Registered protocol family 16 [ 0.191510] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195032] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197094] cpuidle: using governor menu [ 0.198537] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201353] PCI: Using configuration type 1 for base access [ 0.202091] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212013] cryptd: max_cpu_qlen set to 1000 [ 0.213549] ACPI: Added _OSI(Module Device) [ 0.214023] ACPI: Added _OSI(Processor Device) [ 0.214907] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220559] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227544] ACPI: Interpreter enabled [ 0.228021] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230047] ACPI: Using IOAPIC for interrupt routing [ 0.231104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234563] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250499] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267010] acpiphp: Slot [2] registered [ 0.267774] acpiphp: Slot [5] registered [ 0.268048] acpiphp: Slot [6] registered [ 0.269006] acpiphp: Slot [7] registered [ 0.269924] acpiphp: Slot [8] registered [ 0.271049] acpiphp: Slot [9] registered [ 0.271800] acpiphp: Slot [10] registered [ 0.272060] acpiphp: Slot [3] registered [ 0.272897] acpiphp: Slot [4] registered [ 0.274081] acpiphp: Slot [11] registered [ 0.275112] acpiphp: Slot [12] registered [ 0.276059] acpiphp: Slot [13] registered [ 0.277098] acpiphp: Slot [14] registered [ 0.279072] acpiphp: Slot [15] registered [ 0.280071] acpiphp: Slot [16] registered [ 0.281061] acpiphp: Slot [17] registered [ 0.282125] acpiphp: Slot [18] registered [ 0.284085] acpiphp: Slot [19] registered [ 0.285075] acpiphp: Slot [20] registered [ 0.286130] acpiphp: Slot [21] registered [ 0.287084] acpiphp: Slot [22] registered [ 0.289056] acpiphp: Slot [23] registered [ 0.290046] acpiphp: Slot [24] registered [ 0.290951] acpiphp: Slot [25] registered [ 0.291046] acpiphp: Slot [26] registered [ 0.293070] acpiphp: Slot [27] registered [ 0.294043] acpiphp: Slot [28] registered [ 0.295000] acpiphp: Slot [29] registered [ 0.295000] acpiphp: Slot [30] registered [ 0.295000] acpiphp: Slot [31] registered [ 0.297046] PCI host bridge to bus 0000:00 [ 0.299011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309155] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311762] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.313817] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.321010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323032] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.326006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328695] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331741] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333021] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334535] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.358013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.387048] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.407017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.451000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.456016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.485023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494999] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.523049] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.545028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555137] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.562024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.571020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.593021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.605550] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.613019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.617017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.629021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641660] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653042] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.669021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.680363] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.684500] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.687462] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.689518] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.692460] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698125] iommu: Default domain type: Passthrough [ 0.700778] SCSI subsystem initialized [ 0.702301] ACPI: bus type USB registered [ 0.703113] usbcore: registered new interface driver usbfs [ 0.705101] usbcore: registered new interface driver hub [ 0.707336] usbcore: registered new device driver usb [ 0.709438] pps_core: LinuxPPS API ver. 1 registered [ 0.711105] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715108] PTP clock support registered [ 0.717142] EDAC MC: Ver: 3.0.0 [ 0.719147] PCI: Using ACPI for IRQ routing [ 0.721266] NetLabel: Initializing [ 0.723008] NetLabel: domain hash size = 128 [ 0.724023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.727104] NetLabel: unlabeled traffic allowed by default [ 0.729165] vgaarb: loaded [ 0.732274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.733020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741090] clocksource: Switched to clocksource kvm-clock [ 0.882789] VFS: Disk quotas dquot_6.6.0 [ 0.884394] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.886672] *** VALIDATE ramfs *** [ 0.887848] *** VALIDATE hugetlbfs *** [ 0.889394] pnp: PnP ACPI init [ 0.891967] pnp: PnP ACPI: found 6 devices [ 0.908976] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.912350] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.914326] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.916266] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.918417] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.920538] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.923049] NET: Registered protocol family 2 [ 0.925447] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.930536] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.933487] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.938070] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.941327] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953370] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.957591] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.961080] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.965751] NET: Registered protocol family 1 [ 0.968100] RPC: Registered named UNIX socket transport module. [ 0.970023] RPC: Registered udp transport module. [ 0.971585] RPC: Registered tcp transport module. [ 0.973246] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.975580] NET: Registered protocol family 44 [ 0.977318] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.979183] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.980986] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.983380] PCI: CLS 0 bytes, default 64 [ 0.984914] Unpacking initramfs... [ 2.948953] debug: unmapping init [mem 0xffff9e667cc54000-0xffff9e667ffbffff] [ 2.952688] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.954428] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.956672] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.486479] Initialise system trusted keyrings [ 3.487881] Key type blacklist registered [ 3.490439] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.501528] zbud: loaded [ 3.505472] *** VALIDATE nfs *** [ 3.506711] *** VALIDATE nfs4 *** [ 3.509820] pstore: using deflate compression [ 3.515409] Platform Keyring initialized [ 3.697731] NET: Registered protocol family 38 [ 3.699651] Key type asymmetric registered [ 3.703136] Asymmetric key parser 'x509' registered [ 3.708990] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.716759] io scheduler mq-deadline registered [ 3.719577] io scheduler kyber registered [ 3.727747] io scheduler bfq registered [ 3.745591] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.749287] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.752428] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.755513] ACPI: Power Button [PWRF] [ 3.791876] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.802126] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.847769] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.859273] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.882406] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.916465] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.951101] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.956149] Non-volatile memory driver v1.3 [ 3.957793] Linux agpgart interface v0.103 [ 3.994255] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.998227] vda: detected capacity change from 0 to 68911104 [ 4.014940] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.020934] vdb: detected capacity change from 0 to 1073741824 [ 4.055713] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.058609] vdc: detected capacity change from 0 to 2621440000 [ 4.092284] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.094972] vdd: detected capacity change from 0 to 2621440000 [ 4.116257] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.119092] vde: detected capacity change from 0 to 4294967296 [ 4.137282] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.141282] vdf: detected capacity change from 0 to 4294967296 [ 4.155834] libphy: Fixed MDIO Bus: probed [ 4.161988] usbcore: registered new interface driver usbserial_generic [ 4.164158] usbserial: USB Serial support registered for generic [ 4.166865] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.172177] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.174616] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.178330] mousedev: PS/2 mouse device common for all mice [ 4.182982] rtc_cmos 00:05: RTC can wake from S4 [ 4.186520] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.188354] rtc_cmos 00:05: registered as rtc0 [ 4.188417] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.191294] intel_pstate: CPU model not supported [ 4.203847] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.209449] hid: raw HID events driver (C) Jiri Kosina [ 4.217415] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.220851] hpet1: lost 1 rtc interrupts [ 4.221294] usbcore: registered new interface driver usbhid [ 4.229892] usbhid: USB HID core driver [ 4.231705] drop_monitor: Initializing network drop monitor service [ 4.234625] Initializing XFRM netlink socket [ 4.236866] NET: Registered protocol family 10 [ 4.240380] Segment Routing with IPv6 [ 4.242184] NET: Registered protocol family 17 [ 4.244508] mpls_gso: MPLS GSO support [ 4.250921] RAS: Correctable Errors collector initialized. [ 4.253452] AVX version of gcm_enc/dec engaged. [ 4.255231] AES CTR mode by8 optimization enabled [ 4.362785] sched_clock: Marking stable (4362707813, 0)->(5209771872, -847064059) [ 4.367469] registered taskstats version 1 [ 4.371121] Loading compiled-in X.509 certificates [ 4.372839] zswap: loaded using pool lzo/zbud [ 4.410828] Key type big_key registered [ 4.435147] Key type encrypted registered [ 4.437572] ima: No TPM chip found, activating TPM-bypass! [ 4.441595] ima: Allocated hash algorithm: sha1 [ 4.443764] ima: No architecture policies found [ 4.447395] evm: Initialising EVM extended attributes: [ 4.450570] evm: security.selinux [ 4.452757] evm: security.ima [ 4.453972] evm: security.capability [ 4.456803] evm: HMAC attrs: 0x1 [ 4.462428] rtc_cmos 00:05: setting system clock to 2026-03-11 12:38:18 UTC (1773232698) [ 4.468588] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 4.472416] debug: unmapping init [mem 0xffffffffacd82000-0xffffffffad058fff] [ 4.481155] Write protecting the kernel read-only data: 28672k [ 4.487812] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 4.492777] debug: unmapping init [mem 0xffffffffabd14000-0xffffffffabdfffff] [ 4.601732] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.612922] systemd[1]: Detected virtualization kvm. [ 4.614988] systemd[1]: Detected architecture x86-64. [ 4.616947] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.812559] systemd[1]: No hostname configured. [ 4.814485] systemd[1]: Set hostname to . [ 4.816597] random: systemd: uninitialized urandom read (16 bytes read) [ 4.819351] systemd[1]: Initializing machine ID from random generator. [ 4.949258] random: ln: uninitialized urandom read (6 bytes read) [ 5.092165] random: systemd: uninitialized urandom read (16 bytes read) [ 5.100588] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.113862] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.129450] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.093931] device-mapper: uevent: version 1.0.3 [ 6.198647] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 8.231774] random: fast init done [ 8.248065] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.029093] scsi host0: ata_piix [ 9.069200] scsi host1: ata_piix [ 9.070595] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.073341] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.021862] random: crng init done [ 14.028725] random: 7 urandom warning(s) missed due to ratelimiting [ 15.251515] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.129115] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.620144] printk: systemd: 25 output lines suppressed due to ratelimiting [ 19.193778] SELinux: Disabled at runtime. [ 19.349502] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.358651] systemd[1]: Detected virtualization kvm. [ 19.360540] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.074874] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.079220] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.090163] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.099350] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.105436] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.114806] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.126355] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 21.455687] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 22.120576] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.425160] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.496433] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.398935] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.458089] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 28.188902] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 28.740766] NFS: Registering the id_resolver key type [ 28.743380] Key type id_resolver registered [ 28.746169] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg110-server login: [ 57.012668] libcfs: loading out-of-tree module taints kernel. [ 57.033475] Key type ._llcrypt registered [ 57.034675] Key type .llcrypt registered [ 57.078858] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 64.909239] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 65.560250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.569171] alg: No test for adler32 (adler32-zlib) [ 66.628277] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 66.948320] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 68.576180] Key type lgssc registered [ 69.291411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.669059] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.172936] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 99.474790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 99.487106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 100.621777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 100.639665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 100.681453] Lustre: lustre-MDT0000: new disk, initializing [ 100.721496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.733210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 102.326670] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.379697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 108.417820] Lustre: 6479:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 108.433547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 108.435281] Lustre: Skipped 1 previous similar message [ 108.470202] Lustre: lustre-MDT0001: new disk, initializing [ 108.490056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.499410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 108.503146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 110.032425] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 112.506385] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 115.974851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.071828] Lustre: lustre-OST0000: new disk, initializing [ 116.074341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 116.098033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.023119] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.241067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 123.285839] Lustre: lustre-OST0001: new disk, initializing [ 123.287365] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 123.300768] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.955577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 123.959323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 123.964421] Lustre: Skipped 1 previous similar message [ 123.977368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 123.985816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 125.036830] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.035815] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 133.843826] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 141.120863] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing check_logdir /tmp/testlogs/ [ 142.740186] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing yml_node [ 144.399089] Lustre: DEBUG MARKER: Client: 2.17.50.236 [ 145.309596] Lustre: DEBUG MARKER: MDS: 2.17.50.236 [ 146.266646] Lustre: DEBUG MARKER: OSS: 2.17.50.236 [ 146.905243] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Mar 11 08:40:40 EDT 2026 [ 153.004249] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 153.477449] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 169.952610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 169.953383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 169.956488] Lustre: Skipped 2 previous similar messages [ 169.957767] Lustre: Skipped 1 previous similar message [ 171.360415] Lustre: server umount lustre-MDT0000 complete [ 172.725627] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773232866 with bad export cookie 5038189637747559431 [ 172.727066] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 172.730018] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 175.072886] LustreError: 6487:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 175.073026] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 175.073725] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 175.073729] Lustre: Skipped 2 previous similar messages [ 175.078440] LustreError: 6487:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 175.082880] Lustre: Skipped 2 previous similar messages [ 177.120504] LustreError: 6488:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 177.120644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 177.126057] LustreError: 6488:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 177.128032] Lustre: Skipped 2 previous similar messages [ 182.240542] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 182.240679] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 182.244919] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 182.247640] Lustre: Skipped 1 previous similar message [ 187.360346] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 187.361206] LustreError: 9437:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.361303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 187.375465] LustreError: 9437:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 187.458073] Lustre: server umount lustre-MDT0001 complete [ 195.202801] Lustre: server umount lustre-OST0000 complete [ 202.847553] Lustre: server umount lustre-OST0001 complete [ 205.561889] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 209.356785] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 224.306977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.433796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.455225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.505076] Lustre: lustre-MDT0000: new disk, initializing [ 224.557239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.568062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.572892] Lustre: Skipped 1 previous similar message [ 226.333776] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 231.305842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.373342] Lustre: 14944:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.536897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 231.541837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.273915] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 236.993704] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 238.495075] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.316314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.463327] Lustre: lustre-OST0000: new disk, initializing [ 241.465443] Lustre: Skipped 1 previous similar message [ 241.467939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.471318] Lustre: Skipped 2 previous similar messages [ 241.518355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.521639] Lustre: Skipped 1 previous similar message [ 242.626316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.631150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.684643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.886436] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 247.602753] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 249.285922] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 249.385668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 251.190571] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 251.297669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 252.897627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 252.899090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.902733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 252.907104] Lustre: Skipped 1 previous similar message [ 252.910347] Lustre: Skipped 1 previous similar message [ 258.335606] Lustre: server umount lustre-OST0000 complete [ 262.113299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.117121] LustreError: Skipped 1 previous similar message [ 262.119548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.377507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.380635] Lustre: Skipped 9 previous similar messages [ 274.400311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 274.477240] Lustre: server umount lustre-MDT0000 complete [ 275.887305] LustreError: 14938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773232969 with bad export cookie 5038189637747560866 [ 275.890102] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 275.891679] LustreError: 14938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 276.020088] Lustre: server umount lustre-MDT0001 complete [ 277.798713] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 08:42:51 (1773232971) [ 280.317423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.514663] LustreError: 18799:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.542844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 281.943130] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 285.279404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.033930] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 289.405589] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 290.992657] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 293.681873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.183213] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 298.486368] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 311.241833] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 315.364707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.367759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.371689] Lustre: Skipped 3 previous similar messages [ 315.375297] Lustre: Skipped 1 previous similar message [ 320.813819] Lustre: server umount lustre-MDT0000 complete [ 322.439875] LustreError: 21013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233016 with bad export cookie 5038189637747561748 [ 322.443636] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.444555] LustreError: 21013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 322.628461] Lustre: server umount lustre-MDT0001 complete [ 334.576059] Lustre: server umount lustre-OST0000 complete [ 343.584657] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 349.244301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.540698] LustreError: 22848:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.585527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.591717] Lustre: Skipped 2 previous similar messages [ 351.793113] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 357.115043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.672371] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 361.296202] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 364.550875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.616491] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 371.989319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 372.074427] Lustre: lustre-OST0001: new disk, initializing [ 372.077131] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 373.797760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 373.803629] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 373.864917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 374.863919] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 380.132034] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 381.619910] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 393.185443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 393.189395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.195081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 393.198535] Lustre: Skipped 4 previous similar messages [ 397.726474] Lustre: server umount lustre-MDT0000 complete [ 399.191250] LustreError: 25114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233093 with bad export cookie 5038189637747562595 [ 399.195493] LustreError: 25114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 399.195710] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 399.342547] Lustre: server umount lustre-MDT0001 complete [ 410.058040] Lustre: server umount lustre-OST0000 complete [ 421.811325] Lustre: server umount lustre-OST0001 complete [ 428.206027] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 432.303523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.479278] LustreError: 28096:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.488186] LustreError: 28096:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 432.507234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.510390] Lustre: Skipped 3 previous similar messages [ 434.221655] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 437.642804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.287365] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 440.397157] Lustre: 29211:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 442.979021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 445.371578] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 448.991975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.428813] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 454.961083] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 456.394096] Lustre: 31061:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 463.329646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 463.332872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.336179] Lustre: Skipped 3 previous similar messages [ 469.418427] Lustre: server umount lustre-MDT0000 complete [ 469.476655] LustreError: 28810:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.483472] LustreError: 28810:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 470.969286] LustreError: 28077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233165 with bad export cookie 5038189637747563890 [ 470.974832] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.975439] LustreError: 28077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 471.130187] Lustre: server umount lustre-MDT0001 complete [ 482.832188] Lustre: server umount lustre-OST0000 complete [ 494.783587] Lustre: server umount lustre-OST0001 complete [ 498.824072] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 08:46:32 (1773233192) [ 510.625688] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 514.981721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.737347] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 520.444428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.500468] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 523.795482] Lustre: 34955:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 526.767650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 529.394342] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 532.957915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 535.200201] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 543.669676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 545.054866] Lustre: 36808:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 556.002872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.006836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.008190] Lustre: Skipped 6 previous similar messages [ 556.011170] Lustre: Skipped 12 previous similar messages [ 557.739651] Lustre: server umount lustre-MDT0000 complete [ 559.010146] LustreError: 34559:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.016334] LustreError: 34559:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 559.294328] LustreError: 33822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233253 with bad export cookie 5038189637747564996 [ 559.297704] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.304242] LustreError: 33822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 559.442750] Lustre: server umount lustre-MDT0001 complete [ 571.403838] Lustre: server umount lustre-OST0000 complete [ 582.058901] Lustre: server umount lustre-OST0001 complete [ 585.682613] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 08:47:58 (1773233278) [ 592.226626] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 596.497657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.711308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.713558] Lustre: Skipped 7 previous similar messages [ 598.389853] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 602.264936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.103674] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 605.391100] Lustre: 40045:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 608.293195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 611.105806] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 615.070455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.171863] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 627.338990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 634.338826] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 634.342417] Lustre: Skipped 1 previous similar message [ 636.387810] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.393636] Lustre: Skipped 3 previous similar messages [ 641.847578] Lustre: server umount lustre-MDT0000 complete [ 643.691543] LustreError: 38912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233337 with bad export cookie 5038189637747566102 [ 643.696074] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.696561] LustreError: 38912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 649.928405] Lustre: server umount lustre-MDT0001 complete [ 653.715507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.794051] LustreError: 3634:0:(client.c:1428:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e66f69d6680 x1859369314864640/t0(0) o250->MGC192.168.201.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 655.422336] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 658.676849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.586212] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 662.834895] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 671.684665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 673.189917] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 673.260922] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 673.824656] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 681.438836] Lustre: server umount lustre-MDT0000 complete [ 682.674347] LustreError: 38910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233376 with bad export cookie 5038189637747567243 [ 682.679798] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.794457] Lustre: server umount lustre-MDT0001 complete [ 694.409649] Lustre: server umount lustre-OST0000 complete [ 705.058112] Lustre: server umount lustre-OST0001 complete [ 708.238331] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 08:50:01 (1773233401) [ 714.846126] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 718.939167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.098457] LustreError: 46262:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.107682] LustreError: 46262:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 720.725599] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 723.858506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.578130] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 726.666730] Lustre: 47379:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 726.670283] Lustre: 47379:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 729.213205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.643504] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 735.096367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.580463] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 741.155761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 747.831857] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 754.657606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 758.383899] Lustre: server umount lustre-MDT0000 complete [ 759.873366] LustreError: 46999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233453 with bad export cookie 5038189637747568055 [ 759.880430] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 759.886069] LustreError: 46999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 760.016271] Lustre: server umount lustre-MDT0001 complete [ 771.785938] Lustre: server umount lustre-OST0000 complete [ 783.455700] Lustre: server umount lustre-OST0001 complete [ 787.007040] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 08:51:20 (1773233480) [ 788.012599] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 788.679946] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 08:51:22 (1773233482) [ 792.577577] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 795.863745] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 804.554374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.692779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 804.728082] Lustre: lustre-MDT0000: new disk, initializing [ 804.763896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 806.205887] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 809.509910] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 811.820642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 813.576530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 813.587086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 814.265059] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 817.905087] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 818.742289] Lustre: DEBUG MARKER: create test files [ 834.018487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 834.018639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.026635] Lustre: Skipped 8 previous similar messages [ 834.028826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 834.030393] Lustre: Skipped 24 previous similar messages [ 836.381127] Lustre: server umount lustre-OST0000 complete [ 837.905232] Lustre: server umount lustre-MDT0000 complete [ 842.066956] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 843.243046] Key type lgssc unregistered [ 843.377610] LNet: 54770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.379754] LNetError: 54770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.391281] LNet: Removed LNI 192.168.201.110@tcp [ 843.748132] Key type .llcrypt unregistered [ 843.749246] Key type ._llcrypt unregistered [ 854.436256] Key type ._llcrypt registered [ 854.437212] Key type .llcrypt registered [ 854.475248] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 854.745837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.751935] alg: No test for adler32 (adler32-zlib) [ 855.619352] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 855.706549] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 857.288128] Key type lgssc registered [ 857.640981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.332570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.340667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.523923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.872615] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 864.692351] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 866.892504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 867.013045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.128972] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 871.144423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 871.180947] Lustre: 57348:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 871.184601] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 871.184645] Lustre: 57348:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 871.194750] Lustre: 57348:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 871.197982] Lustre: 57348:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 871.201300] Lustre: 57348:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 871.205068] Lustre: 57348:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 874.320171] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 876.442129] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 881.633547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.633572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 881.637982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.763832] Lustre: server umount lustre-OST0000 complete [ 886.310335] Lustre: server umount lustre-MDT0000 complete [ 890.302806] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 891.390698] Key type lgssc unregistered [ 891.537576] LNet: 58826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.540709] LNetError: 58826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.547466] LNet: Removed LNI 192.168.201.110@tcp [ 891.865973] Key type .llcrypt unregistered [ 891.867567] Key type ._llcrypt unregistered [ 904.620487] Key type ._llcrypt registered [ 904.622145] Key type .llcrypt registered [ 904.662629] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 905.207781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.224441] alg: No test for adler32 (adler32-zlib) [ 906.092062] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 906.188237] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 907.776242] Key type lgssc registered [ 908.214054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 911.080023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 911.089848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.300149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.894718] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 915.924253] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 918.346606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 918.464905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.523828] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 922.526485] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 922.602156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 922.627407] Lustre: 61508:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 922.632045] Lustre: 61508:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 922.634655] Lustre: 61508:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 922.637979] Lustre: 61508:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 922.643414] Lustre: 61508:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 922.645956] Lustre: 61508:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 925.636279] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 930.516821] Lustre: server umount lustre-OST0000 complete [ 941.070263] Lustre: server umount lustre-MDT0000 complete [ 944.933129] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 945.970704] Key type lgssc unregistered [ 946.095327] LNet: 62937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.097738] LNetError: 62937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.108265] LNet: Removed LNI 192.168.201.110@tcp [ 946.437907] Key type .llcrypt unregistered [ 946.439355] Key type ._llcrypt unregistered [ 952.364290] Key type ._llcrypt registered [ 952.365702] Key type .llcrypt registered [ 952.400953] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 958.868378] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 959.338383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 959.347442] alg: No test for adler32 (adler32-zlib) [ 960.213933] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 960.299607] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 961.880161] Key type lgssc registered [ 962.238341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.049489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.056292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.141579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 976.153443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 976.187971] Lustre: lustre-MDT0000: new disk, initializing [ 976.215502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.221647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 977.524174] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 982.017589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.060053] Lustre: 66032:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 982.081512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 982.083181] Lustre: Skipped 1 previous similar message [ 982.117455] Lustre: lustre-MDT0001: new disk, initializing [ 982.152425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.169691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 982.174164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 983.721378] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 986.970285] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 988.331321] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 990.959783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.089553] Lustre: lustre-OST0000: new disk, initializing [ 991.091391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 991.122437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.204941] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 993.533792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 993.539192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 993.583193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 996.395378] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 997.791624] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 997.863667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 999.181387] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 999.257750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1004.001467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.002073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.004088] LustreError: Skipped 1 previous similar message [ 1004.008666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.011777] Lustre: Skipped 1 previous similar message [ 1009.125075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.127428] Lustre: Skipped 1 previous similar message [ 1014.240147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1014.244123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.247325] Lustre: Skipped 1 previous similar message [ 1014.303213] Lustre: server umount lustre-OST0000 complete [ 1018.337069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1018.340515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.345466] Lustre: Skipped 1 previous similar message [ 1018.348015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.362280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.787972] Lustre: server umount lustre-MDT0000 complete [ 1023.236429] LustreError: 66809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233717 with bad export cookie 14770054756852244913 [ 1023.240597] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.240683] LustreError: 66809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1023.368958] Lustre: server umount lustre-MDT0001 complete [ 1026.658757] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 08:55:20 (1773233720) [ 1030.614478] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1043.366606] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1046.853688] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1058.866168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.957580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1058.970953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1059.012106] Lustre: lustre-MDT0000: new disk, initializing [ 1059.039171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.045296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1060.352329] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1064.623789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.663311] Lustre: 73582:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1064.683455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1064.686460] Lustre: Skipped 1 previous similar message [ 1064.721321] Lustre: lustre-MDT0001: new disk, initializing [ 1064.750808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1064.761075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1064.764255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1066.114374] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1069.105175] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.491860] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1073.061637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1073.170329] Lustre: lustre-OST0000: new disk, initializing [ 1073.172707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1073.200338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.364136] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1078.265478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1078.269232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1078.307990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1078.888039] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1080.507704] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1080.624254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1082.216522] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1082.301731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1083.365673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1083.365704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.369383] LustreError: Skipped 1 previous similar message [ 1083.375119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.378446] Lustre: Skipped 3 previous similar messages [ 1085.080204] Lustre: server umount lustre-OST0000 complete [ 1088.486586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1088.495590] Lustre: Skipped 2 previous similar messages [ 1092.702223] Lustre: server umount lustre-MDT0000 complete [ 1094.185877] LustreError: 75322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233788 with bad export cookie 14770054756852245795 [ 1094.188882] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.191342] LustreError: 75322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1094.334348] Lustre: server umount lustre-MDT0001 complete [ 1097.326335] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 08:56:30 (1773233790) [ 1099.898492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.082272] LustreError: 77583:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.091328] LustreError: 77583:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1100.108942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.795882] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1105.194681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.034996] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1109.451168] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1111.008392] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1113.793615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1116.085350] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1118.150365] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1119.549226] Lustre: 79917:0:(genops.c:1793:obd_export_evict_by_uuid()) MGS: evicting 28d327ec-9c61-4909-8cae-835ea5854682 at adminstrative request [ 1120.229332] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.233751] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1130.469166] Lustre: MGS: Client 28d327ec-9c61-4909-8cae-835ea5854682 (at 0@lo) reconnecting [ 1130.472307] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xccf9c68bd73f2887 to 0xccf9c68bd73f2b12 [ 1130.476655] Lustre: MGC192.168.201.110@tcp: Connection restored to 0@lo (at 0@lo) [ 1152.099134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1152.171200] Lustre: lustre-OST0001: new disk, initializing [ 1152.173966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1152.212131] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1152.215573] Lustre: Skipped 2 previous similar messages [ 1154.283592] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1157.409686] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1158.350958] Lustre: server umount lustre-OST0001 complete [ 1159.663512] LustreError: 79174:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.664323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.673214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.675136] Lustre: Skipped 3 previous similar messages [ 1164.773146] LustreError: 79176:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.777935] LustreError: 79176:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1165.648231] Lustre: server umount lustre-OST0000 complete [ 1180.129886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.133433] Lustre: Skipped 7 previous similar messages [ 1181.152140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1181.223847] Lustre: server umount lustre-MDT0000 complete [ 1182.580524] LustreError: 77566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233876 with bad export cookie 14770054756852247314 [ 1182.585635] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1182.586646] LustreError: 77566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1182.707934] Lustre: server umount lustre-MDT0001 complete [ 1185.593690] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 08:57:58 (1773233878) [ 1186.134431] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1186.731915] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 08:58:00 (1773233880) [ 1187.263058] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1187.808051] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 08:58:01 (1773233881) [ 1188.348143] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1188.887516] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 08:58:02 (1773233882) [ 1189.442505] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 1190.056515] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 08:58:03 (1773233883) [ 1190.606212] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1192.784113] loop: module loaded [ 1192.789229] loop0: detected capacity change from 0 to 4294967296 [ 1192.803740] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1193.328524] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1194.286503] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1195.796156] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1204.638909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.652456] LustreError: 84293:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1204.654825] LustreError: 84293:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1204.656963] LustreError: 84293:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1204.659281] LustreError: 84293:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1210.851778] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1214.406839] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1226.691027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.794670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1226.805756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1226.844279] Lustre: lustre-MDT0000: new disk, initializing [ 1226.869392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.877629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1228.157052] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1232.523215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.564133] Lustre: 87117:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1232.662468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1233.998762] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1237.084877] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1238.261642] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1240.470741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1240.797522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1240.838716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1242.429296] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1245.477921] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1246.780620] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1246.849426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1248.139566] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1248.214503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1251.299273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1251.299705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.300875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.301529] LustreError: Skipped 1 previous similar message [ 1251.307312] Lustre: Skipped 4 previous similar messages [ 1254.991363] Lustre: server umount lustre-OST0000 complete [ 1262.445107] Lustre: server umount lustre-MDT0000 complete [ 1263.734688] LustreError: 87109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773233957 with bad export cookie 14770054756852247895 [ 1263.738264] LustreError: 87109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1263.739305] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.859848] Lustre: server umount lustre-MDT0001 complete [ 1266.600168] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 08:59:19 (1773233959) [ 1268.833061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.988238] LustreError: 91122:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.357341] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1273.379860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.919137] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1276.971959] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1278.118884] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1280.415954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1282.402629] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1284.286687] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1285.772470] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1296.087987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 1297.521237] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1297.604407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1299.852058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.924669] Lustre: lustre-OST0001: new disk, initializing [ 1299.926108] Lustre: Skipped 2 previous similar messages [ 1299.927729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1299.929260] Lustre: Skipped 3 previous similar messages [ 1302.011511] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1305.311577] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1306.749879] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1307.674287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1307.680414] Lustre: Skipped 2 previous similar messages [ 1307.684676] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1307.732777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1307.868987] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1309.167218] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1309.233899] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1319.454476] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1324.727669] Lustre: Failing over lustre-MDT0000 [ 1325.025265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1325.027634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.031092] Lustre: Skipped 2 previous similar messages [ 1325.033059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.034829] Lustre: Skipped 5 previous similar messages [ 1325.063302] Lustre: server umount lustre-MDT0000 complete [ 1325.990068] LustreError: 93545:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.994700] LustreError: 93545:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1336.223112] LustreError: 93545:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.229272] LustreError: 93545:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 11 previous similar messages [ 1339.897814] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1339.899274] LDISKFS-fs (dm-0): recovery complete [ 1339.902253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.937567] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.067255] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1341.278197] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1341.344532] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1341.347171] Lustre: 91118:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1345.506513] Lustre: 91123:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1345.509768] Lustre: 91123:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 1345.514688] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1345.530519] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1345.848091] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1346.162323] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1346.165733] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 1 previous similar message [ 1346.480109] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1346.482732] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 1 previous similar message [ 1346.802466] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1346.805541] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 8 previous similar messages [ 1347.433984] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1347.436561] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 3 previous similar messages [ 1347.752086] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1347.754462] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 3 previous similar messages [ 1349.008745] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1349.011914] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 6 previous similar messages [ 1349.641048] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1349.643655] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 6 previous similar messages [ 1349.960110] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1349.962559] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 6 previous similar messages [ 1353.090794] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1353.094336] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 1353.722912] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1353.725931] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 12 previous similar messages [ 1354.040116] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1354.042404] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 12 previous similar messages [ 1361.226393] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1361.229200] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 25 previous similar messages [ 1361.858398] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1361.862403] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 25 previous similar messages [ 1362.176092] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1362.178908] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 25 previous similar messages [ 1377.482154] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1377.485261] Lustre: 96875:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 51 previous similar messages [ 1378.114108] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1378.117238] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 51 previous similar messages [ 1378.432113] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1378.434733] LustreError: 96875:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 51 previous similar messages [ 1401.216166] Lustre: 96875:0:(ldlm_lib.c:2376:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1401.218756] Lustre: 96875:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1401.221145] Lustre: 96875:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0d127892-6306-4f52-bec9-0271204519c1@192.168.201.10@tcp [ 1401.224389] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1401.226528] Lustre: 96875:0:(ldlm_lib.c:1896:abort_req_replay_queue()) @@@ aborted: req@00000000706dbdf7 x1859370244704128/t0(8589934775) o101->0d127892-6306-4f52-bec9-0271204519c1@192.168.201.10@tcp:335/0 lens 592/0 e 0 to 0 dl 1773234105 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1401.234178] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1401.239763] Lustre: lustre-MDT0000: Denying connection for new client 0d127892-6306-4f52-bec9-0271204519c1 (at 192.168.201.10@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 1401.239938] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1401.256362] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1401.260313] Lustre: Skipped 2 previous similar messages [ 1401.261343] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1401.279490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 1401.279490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 1411.553256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1411.555643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.562464] Lustre: Skipped 3 previous similar messages [ 1413.182902] Lustre: server umount lustre-OST0000 complete [ 1417.188840] LustreError: 96242:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.193623] LustreError: 96242:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 1420.506809] Lustre: server umount lustre-OST0001 complete [ 1436.128107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1436.193414] Lustre: server umount lustre-MDT0000 complete [ 1437.420103] LustreError: 94191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234131 with bad export cookie 14770054756852299023 [ 1437.422794] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.423899] LustreError: 94191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1437.563744] Lustre: server umount lustre-MDT0001 complete [ 1440.486321] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:02:13 (1773234133) [ 1443.058420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1443.062074] Lustre: *** cfs_fail_loc=197, val=0*** [ 1458.528676] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1463.648365] LustreError: 98846:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.110@tcp: failed processing log, type 4: rc = -110 [ 1468.776153] Lustre: *** cfs_fail_loc=197, val=0*** [ 1468.777349] Lustre: Skipped 65 previous similar messages [ 1489.248530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.250320] Lustre: Skipped 7 previous similar messages [ 1491.108070] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1492.890797] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1493.585314] Lustre: server umount lustre-OST0000 complete [ 1496.382373] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:03:09 (1773234189) [ 1510.195030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.363643] LustreError: 100350:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.726815] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1514.954059] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1516.985813] LustreError: 100382:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1516.991549] LustreError: 100382:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1516.996426] LustreError: 100382:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1517.131416] Lustre: server umount lustre-MDT0000 complete [ 1527.514253] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1530.545085] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1542.860358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.950351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1542.960605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1542.996497] Lustre: lustre-MDT0000: new disk, initializing [ 1543.024274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1544.364247] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1548.547263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.590083] Lustre: 104402:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1548.691206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1549.939648] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1552.983238] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1554.183377] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1556.514935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1558.465969] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1559.275718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1561.503576] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1562.814936] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1562.884448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1564.148358] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1564.217795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1569.762805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1569.764179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.765147] LustreError: Skipped 4 previous similar messages [ 1569.769372] Lustre: Skipped 5 previous similar messages [ 1569.771659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.773921] Lustre: Skipped 13 previous similar messages [ 1570.970833] Lustre: server umount lustre-OST0000 complete [ 1578.404310] Lustre: server umount lustre-MDT0000 complete [ 1579.676076] LustreError: 104394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234273 with bad export cookie 14770054756852309663 [ 1579.679521] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.679554] LustreError: 104394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1579.789189] Lustre: server umount lustre-MDT0001 complete [ 1582.611645] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:04:36 (1773234276) [ 1585.756947] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1586.897568] Key type lgssc unregistered [ 1587.032407] LNet: 108459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.034940] LNetError: 108459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.046372] LNet: Removed LNI 192.168.201.110@tcp [ 1587.391996] Key type .llcrypt unregistered [ 1587.393849] Key type ._llcrypt unregistered [ 1593.131891] Key type ._llcrypt registered [ 1593.132953] Key type .llcrypt registered [ 1593.179903] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1599.550182] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1600.008058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.016979] alg: No test for adler32 (adler32-zlib) [ 1600.866553] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1600.949200] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1602.528135] Key type lgssc registered [ 1602.903178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.860975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.866193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.966685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1616.981311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1617.029174] Lustre: lustre-MDT0000: new disk, initializing [ 1617.054786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.061866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1618.460117] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1622.688195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1622.791473] Lustre: lustre-OST0000: new disk, initializing [ 1622.793765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1622.795579] Lustre: Skipped 1 previous similar message [ 1622.818505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.746784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1623.750126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1623.759223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1624.713847] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1628.954076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1629.018495] Lustre: lustre-OST0001: new disk, initializing [ 1629.020397] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1629.050038] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1629.547675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1629.552767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1629.564943] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1630.969865] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1639.905785] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.906540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1639.910034] Lustre: Skipped 1 previous similar message [ 1641.204815] Lustre: server umount lustre-MDT0000 complete [ 1643.320201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1644.873568] LustreError: 111758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234338 with bad export cookie 3855572574258050125 [ 1644.875242] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1644.879468] LustreError: 111758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1644.912868] Lustre: server umount lustre-OST0000 complete [ 1646.152107] Lustre: server umount lustre-OST0001 complete [ 1648.920073] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 09:05:42 (1773234342) [ 1651.759310] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1659.095757] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:05:52 (1773234352) [ 1671.178626] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 1674.139159] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1685.458420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.540577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1685.551082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1685.586327] Lustre: lustre-MDT0000: new disk, initializing [ 1685.609124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.615378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1686.816452] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1690.837396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.879452] Lustre: 120070:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.914411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1690.915252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1690.916167] Lustre: Skipped 1 previous similar message [ 1690.918171] Lustre: Skipped 1 previous similar message [ 1690.957355] Lustre: lustre-MDT0001: new disk, initializing [ 1690.989378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1691.001152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1691.004324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1692.235492] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1695.301126] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1696.536472] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1698.800653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.721322] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1703.413638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1703.417785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1703.449548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1703.706850] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1705.051259] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1705.135582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1706.420419] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1706.508291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1708.518347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.518927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.519652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1708.522032] Lustre: Skipped 1 previous similar message [ 1708.528395] LustreError: Skipped 1 previous similar message [ 1709.209250] Lustre: server umount lustre-OST0000 complete [ 1711.584843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1711.587146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.591062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.593755] Lustre: Skipped 1 previous similar message [ 1713.633983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.634290] Lustre: server umount lustre-MDT0000 complete [ 1717.845164] LustreError: 120061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234411 with bad export cookie 3855572574258051336 [ 1717.848136] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1717.850034] LustreError: 120061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1717.963911] Lustre: server umount lustre-MDT0001 complete [ 1720.740164] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:06:54 (1773234414) [ 1722.972773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.143427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.145270] Lustre: Skipped 1 previous similar message [ 1724.460727] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1727.261035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.758621] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1730.580729] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1731.775126] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1734.053771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.963883] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1737.769689] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1755.524106] Lustre: *** cfs_fail_loc=159, val=0*** [ 1771.952297] Lustre: lustre-MDT0000: Client 6ccfd3ad-8994-4c5d-b869-a98521f8ae2d (at 192.168.201.10@tcp) reconnecting [ 1772.510786] Lustre: *** cfs_fail_loc=159, val=0*** [ 1772.512360] Lustre: Skipped 5 previous similar messages [ 1788.844142] Lustre: lustre-MDT0000: Client 6ccfd3ad-8994-4c5d-b869-a98521f8ae2d (at 192.168.201.10@tcp) reconnecting [ 1803.744866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.747062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.751188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.754305] Lustre: Skipped 2 previous similar messages [ 1809.806066] Lustre: server umount lustre-OST0000 complete [ 1814.496848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1814.500921] LustreError: Skipped 1 previous similar message [ 1814.503333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.508488] Lustre: Skipped 1 previous similar message [ 1814.511530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1814.514465] Lustre: Skipped 2 previous similar messages [ 1817.170884] Lustre: server umount lustre-MDT0000 complete [ 1818.390191] LustreError: 125684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234512 with bad export cookie 3855572574258052218 [ 1818.393852] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.396221] LustreError: 125684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1818.506323] Lustre: server umount lustre-MDT0001 complete [ 1822.161935] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 1823.142749] Key type lgssc unregistered [ 1823.264313] LNet: 127510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.266702] LNetError: 127510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.272301] LNet: Removed LNI 192.168.201.110@tcp [ 1823.558294] Key type .llcrypt unregistered [ 1823.559347] Key type ._llcrypt unregistered [ 1832.730143] Key type ._llcrypt registered [ 1832.731127] Key type .llcrypt registered [ 1832.774532] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 1833.165869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.224843] alg: No test for adler32 (adler32-zlib) [ 1834.087813] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 1834.196141] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 1835.785509] Key type lgssc registered [ 1836.301468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1841.650114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1841.673358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.093464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.541898] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1853.422061] LustreError: 128887:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.436305] LustreError: 128887:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1854.246297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.608360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1857.897980] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1862.770574] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.940522] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1872.029340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1872.407819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.243868] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 1878.836239] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1888.812112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1895.771311] Lustre: *** cfs_fail_loc=159, val=0*** [ 1911.214054] Lustre: lustre-MDT0000: Client 5cc13aa6-2e08-47f2-a2fa-5dc6c4408fbb (at 192.168.201.10@tcp) reconnecting [ 1912.171216] Lustre: *** cfs_fail_loc=159, val=0*** [ 1912.174804] Lustre: Skipped 2 previous similar messages [ 1928.627653] Lustre: lustre-MDT0001: Client 5cc13aa6-2e08-47f2-a2fa-5dc6c4408fbb (at 192.168.201.10@tcp) reconnecting [ 1929.357804] Lustre: *** cfs_fail_loc=159, val=0*** [ 1945.528414] Lustre: lustre-MDT0001: Client 5cc13aa6-2e08-47f2-a2fa-5dc6c4408fbb (at 192.168.201.10@tcp) reconnecting [ 1946.847112] Lustre: 131612:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1947.754375] Lustre: *** cfs_fail_loc=159, val=0*** [ 1947.756041] Lustre: Skipped 4 previous similar messages [ 1963.440283] Lustre: lustre-MDT0000: Client dde23ac0-4b1e-4a2e-a87c-f7b5e2d52bb7 (at 192.168.201.10@tcp) reconnecting [ 1964.063850] Lustre: *** cfs_fail_loc=159, val=0*** [ 1964.067252] Lustre: Skipped 14 previous similar messages [ 1979.310226] Lustre: lustre-MDT0000: Client dde23ac0-4b1e-4a2e-a87c-f7b5e2d52bb7 (at 192.168.201.10@tcp) reconnecting [ 1980.006192] Lustre: 131908:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1996.259101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.260198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1996.260227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1996.263717] Lustre: Skipped 1 previous similar message [ 1997.359276] Lustre: server umount lustre-OST0000 complete [ 2001.377682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.378555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.383149] Lustre: Skipped 1 previous similar message [ 2001.385082] Lustre: Skipped 1 previous similar message [ 2006.498358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.501126] Lustre: Skipped 2 previous similar messages [ 2011.617355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.620682] Lustre: Skipped 1 previous similar message [ 2013.152171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2013.217421] Lustre: server umount lustre-MDT0000 complete [ 2014.596423] LustreError: 131857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234708 with bad export cookie 15372485675330781535 [ 2014.600264] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.600707] LustreError: 131857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2014.722202] Lustre: server umount lustre-MDT0001 complete [ 2018.550780] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2019.697727] Key type lgssc unregistered [ 2019.837464] LNet: 132967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.841564] LNetError: 132967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.849350] LNet: Removed LNI 192.168.201.110@tcp [ 2020.207821] Key type .llcrypt unregistered [ 2020.209364] Key type ._llcrypt unregistered [ 2031.896303] Key type ._llcrypt registered [ 2031.899085] Key type .llcrypt registered [ 2031.942595] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2032.421761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2032.449092] alg: No test for adler32 (adler32-zlib) [ 2033.348947] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2033.467070] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2035.064163] Key type lgssc registered [ 2035.532598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2038.812261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2038.823732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.014499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.613252] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2045.266412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.428314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.112519] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2049.587239] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2051.374247] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2054.389537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.547274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.593491] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2057.003615] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2059.252139] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2065.836474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2072.360584] Lustre: 136612:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2073.194709] Lustre: 136660:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2073.836292] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2075.390928] Lustre: 136805:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2075.823570] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2075.826312] Lustre: Skipped 1 previous similar message [ 2076.227076] Lustre: 136904:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2079.511001] Lustre: server umount lustre-OST0000 complete [ 2081.250326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2081.253537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.260630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.263032] Lustre: Skipped 1 previous similar message [ 2086.369632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.372705] Lustre: Skipped 1 previous similar message [ 2090.507412] Lustre: server umount lustre-MDT0000 complete [ 2091.489406] LustreError: 134341:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.499886] LustreError: 134341:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2092.347356] LustreError: 135959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234786 with bad export cookie 14242385504622542370 [ 2092.351591] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.353789] LustreError: 135959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2102.811147] Lustre: server umount lustre-MDT0001 complete [ 2107.722267] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2109.049411] Key type lgssc unregistered [ 2109.207670] LNet: 137940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.213353] LNetError: 137940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.223412] LNet: Removed LNI 192.168.201.110@tcp [ 2109.600352] Key type .llcrypt unregistered [ 2109.602537] Key type ._llcrypt unregistered [ 2121.238113] Key type ._llcrypt registered [ 2121.240182] Key type .llcrypt registered [ 2121.293774] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2121.712152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.810047] alg: No test for adler32 (adler32-zlib) [ 2122.681371] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2122.782445] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2124.368173] Key type lgssc registered [ 2124.818492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2128.112017] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2128.122203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.327098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.009668] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2134.969554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.202383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.970874] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2139.558920] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2141.267780] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2144.472843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.666211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.707045] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2147.336347] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2149.791306] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2154.922543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2158.852118] hrtimer: interrupt took 24639914 ns [ 2162.671390] Lustre: *** cfs_fail_loc=159, val=0*** [ 2178.311383] Lustre: lustre-MDT0000: Client 8167c81d-310d-493c-9be2-08a0afb1f97d (at 192.168.201.10@tcp) reconnecting [ 2184.170331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2184.173057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.188123] LustreError: Skipped 2 previous similar messages [ 2184.201278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.697730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.705920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.647669] Lustre: server umount lustre-OST0000 complete [ 2193.275917] LustreError: 138627:0:(client.c:1418:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000019dce573 x1859371470450560/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/224 e 0 to 0 dl 1773234894 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'' uid:0 gid:0 projid:4294967295 [ 2195.940643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.968347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.972178] Lustre: Skipped 1 previous similar message [ 2199.595958] Lustre: server umount lustre-MDT0000 complete [ 2201.057657] LustreError: 139315:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.090274] LustreError: 139315:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2205.052161] LustreError: 140940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773234899 with bad export cookie 7650551992756657644 [ 2205.056147] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.072183] LustreError: 140940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2205.697610] Lustre: server umount lustre-MDT0001 complete [ 2222.272421] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2225.331755] Key type lgssc unregistered [ 2225.577925] LNet: 142646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.585636] LNetError: 142646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.602461] LNet: Removed LNI 192.168.201.110@tcp [ 2226.322578] Key type .llcrypt unregistered [ 2226.325185] Key type ._llcrypt unregistered [ 2247.050473] Key type ._llcrypt registered [ 2247.053053] Key type .llcrypt registered [ 2247.149837] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2248.099574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2248.127387] alg: No test for adler32 (adler32-zlib) [ 2249.119970] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2249.335919] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2250.992177] Key type lgssc registered [ 2251.975346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.783910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.799687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.225960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.568852] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2270.691185] LustreError: 144046:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.710167] LustreError: 144046:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2270.722659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.105909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2274.864122] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2278.566679] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2281.030364] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2286.497726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.793559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.446609] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2295.861789] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2296.069250] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2305.309901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2325.304879] Lustre: lustre-OST0000: evicting 240bacf9-98e2-49a1-a4f1-754ad7a8e1ae (at 192.168.201.10@tcp) by administrative request [ 2325.316917] Lustre: Skipped 1 previous similar message [ 2352.614329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2352.624128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.642183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.708683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.710518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.715860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.687264] Lustre: server umount lustre-OST0000 complete [ 2361.827945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.830383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.841963] Lustre: Skipped 1 previous similar message [ 2361.858908] Lustre: Skipped 2 previous similar messages [ 2363.880795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2363.888929] Lustre: Skipped 1 previous similar message [ 2372.066630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.075056] Lustre: Skipped 3 previous similar messages [ 2376.160139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2376.279077] Lustre: server umount lustre-MDT0000 complete [ 2377.185620] LustreError: 145112:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.207290] LustreError: 145112:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2378.941739] LustreError: 144025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235072 with bad export cookie 13301694680815964958 [ 2378.951052] LustreError: 144025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2378.954259] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.186699] Lustre: server umount lustre-MDT0001 complete [ 2386.946285] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2388.573648] Key type lgssc unregistered [ 2388.778979] LNet: 147782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.784930] LNetError: 147782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.793257] LNet: Removed LNI 192.168.201.110@tcp [ 2389.290473] Key type .llcrypt unregistered [ 2389.293058] Key type ._llcrypt unregistered [ 2449.783525] Key type ._llcrypt registered [ 2449.786332] Key type .llcrypt registered [ 2449.870415] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2450.868406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2450.896087] alg: No test for adler32 (adler32-zlib) [ 2451.907563] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2452.090585] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2453.736775] Key type lgssc registered [ 2454.605768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2461.434952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2461.447722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.871366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.404906] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2472.419501] LustreError: 150252:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.436803] LustreError: 150252:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2473.929214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.310580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2477.942647] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2482.490319] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2486.302959] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2492.573993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2492.850634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.481374] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2498.999734] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2503.506980] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2507.180719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2517.478559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.487298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2517.493244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2517.504282] Lustre: Skipped 1 previous similar message [ 2521.789373] Lustre: server umount lustre-OST0000 complete [ 2525.667149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2525.682880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.701924] Lustre: Skipped 1 previous similar message [ 2525.719779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.713784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.724396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.319739] Lustre: server umount lustre-MDT0000 complete [ 2532.834482] LustreError: 151874:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.856944] LustreError: 151874:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2534.880091] LustreError: 150235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235228 with bad export cookie 17695000191226052006 [ 2534.896808] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2534.900291] LustreError: 150235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2535.166692] Lustre: server umount lustre-MDT0001 complete [ 2544.780230] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2547.135118] Key type lgssc unregistered [ 2547.398068] LNet: 153452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2547.403961] LNetError: 153452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.455224] LNet: Removed LNI 192.168.201.110@tcp [ 2549.258169] Key type .llcrypt unregistered [ 2549.259710] Key type ._llcrypt unregistered [ 2568.576690] Key type ._llcrypt registered [ 2568.579404] Key type .llcrypt registered [ 2568.673608] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2569.603742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.628641] alg: No test for adler32 (adler32-zlib) [ 2570.652147] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2570.899499] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2572.578400] Key type lgssc registered [ 2573.540221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2585.959944] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 09:21:18 (1773235278) [ 2598.150081] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 2606.044568] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2634.286734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2634.319030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.507894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2635.540869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2635.619681] Lustre: lustre-MDT0000: new disk, initializing [ 2635.689653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.704264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2639.203643] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2647.873826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.999648] Lustre: 157668:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2648.057813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2648.062864] Lustre: Skipped 1 previous similar message [ 2648.126849] Lustre: lustre-MDT0001: new disk, initializing [ 2648.221397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.255674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2648.272491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2651.309991] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2657.262497] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2659.858309] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2664.962815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2665.215351] Lustre: lustre-OST0000: new disk, initializing [ 2665.224831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2665.305943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.906660] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2671.137325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2671.150612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2671.244763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2676.195867] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2686.435313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.440263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2686.448367] Lustre: Skipped 1 previous similar message [ 2691.564788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2691.569451] Lustre: Skipped 2 previous similar messages [ 2696.693674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2696.697931] Lustre: Skipped 1 previous similar message [ 2698.208416] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2698.342561] Lustre: server umount lustre-OST0000 complete [ 2701.794094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.796165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.803294] Lustre: Skipped 1 previous similar message [ 2706.916462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2706.922341] Lustre: Skipped 2 previous similar messages [ 2707.636786] Lustre: server umount lustre-MDT0000 complete [ 2710.346584] LustreError: 157660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235404 with bad export cookie 18163749480671309804 [ 2710.348676] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2710.355235] LustreError: 157660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2710.594668] Lustre: server umount lustre-MDT0001 complete [ 2719.090916] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2721.229668] Key type lgssc unregistered [ 2721.480889] LNet: 161185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.491308] LNetError: 161185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.510325] LNet: Removed LNI 192.168.201.110@tcp [ 2722.188713] Key type .llcrypt unregistered [ 2722.191094] Key type ._llcrypt unregistered [ 2743.130293] Key type ._llcrypt registered [ 2743.132860] Key type .llcrypt registered [ 2743.222769] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2744.006536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2744.129368] alg: No test for adler32 (adler32-zlib) [ 2745.187514] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2745.437665] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2747.128162] Key type lgssc registered [ 2748.205743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2755.284866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2755.298614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.726917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.625489] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2766.311455] LustreError: 162588:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.328764] LustreError: 162588:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2767.310356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.554055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2770.556909] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2774.488953] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2777.347942] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2782.946961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2783.198716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.010327] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2792.785477] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2801.656618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2808.507965] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2808.519032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.526810] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2808.532160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.543824] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2809.115509] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2809.131460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.144363] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2809.149878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.157905] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2809.786467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.320732] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2810.325661] Lustre: Skipped 1 previous similar message [ 2810.336992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.346515] Lustre: Skipped 1 previous similar message [ 2810.350852] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2810.355376] Lustre: Skipped 1 previous similar message [ 2810.357865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.368038] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2810.372080] Lustre: Skipped 1 previous similar message [ 2811.003359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.657651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.252642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.923146] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2812.926304] Lustre: Skipped 3 previous similar messages [ 2812.939246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.946823] Lustre: Skipped 3 previous similar messages [ 2812.950946] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2812.954182] Lustre: Skipped 3 previous similar messages [ 2812.956782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.963240] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2812.966290] Lustre: Skipped 3 previous similar messages [ 2813.566897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.181931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.817606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.445457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.190469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.808185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.345973] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2817.348983] Lustre: Skipped 6 previous similar messages [ 2817.369850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.376374] Lustre: Skipped 6 previous similar messages [ 2817.381069] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2817.386038] Lustre: Skipped 6 previous similar messages [ 2817.389974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.398438] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2817.402130] Lustre: Skipped 6 previous similar messages [ 2818.054614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.682943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.347225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.012993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.657627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.239067] LustreError: 162607:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2821.250098] LustreError: 162607:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2821.250423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.900881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.524417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.163980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.820571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.448161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.054289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.698134] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2825.701150] Lustre: Skipped 12 previous similar messages [ 2825.710181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.715694] Lustre: Skipped 12 previous similar messages [ 2825.718683] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2825.722279] Lustre: Skipped 12 previous similar messages [ 2825.727869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.736642] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2825.743278] Lustre: Skipped 12 previous similar messages [ 2826.411901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.072644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.921722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2834.924201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2834.926448] LustreError: Skipped 4 previous similar messages [ 2836.418515] Lustre: server umount lustre-OST0000 complete [ 2839.521679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2839.525891] LustreError: Skipped 3 previous similar messages [ 2839.528965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.594139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.594995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.605976] Lustre: Skipped 4 previous similar messages [ 2844.876945] Lustre: server umount lustre-MDT0000 complete [ 2847.218563] LustreError: 162568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235541 with bad export cookie 7202354650880503618 [ 2847.223775] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.565276] Lustre: server umount lustre-MDT0001 complete [ 2864.604115] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 2866.416796] Key type lgssc unregistered [ 2866.631947] LNet: 167666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.636408] LNetError: 167666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2866.647468] LNet: Removed LNI 192.168.201.110@tcp [ 2867.086405] Key type .llcrypt unregistered [ 2867.089222] Key type ._llcrypt unregistered [ 2882.900893] Key type ._llcrypt registered [ 2882.902928] Key type .llcrypt registered [ 2882.961954] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 2883.708387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.723485] alg: No test for adler32 (adler32-zlib) [ 2884.633994] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 2884.765344] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 2886.376201] Key type lgssc registered [ 2887.054054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.290414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2892.310927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.598928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.211439] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2901.876804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.190509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2905.547850] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2908.723716] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2910.838840] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2915.228437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.512071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.579775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:243 to 0x280000401:289) [ 2919.665994] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2923.063692] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2931.321762] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2932.469178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.481347] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2932.491740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.503727] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2932.506821] LustreError: 169062:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2935.727424] Lustre: lustre-OST0000: Not available for connect from 192.168.201.10@tcp (stopping) [ 2938.339686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.339749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.341759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.355052] Lustre: Skipped 1 previous similar message [ 2940.061716] Lustre: server umount lustre-OST0000 complete [ 2952.544340] LustreError: 169062:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2970.526721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2970.834152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.848495] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 2978.031389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.043397] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.045720] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2979.437305] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2980.789256] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2983.564692] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b958781c800.ost_server_uuid 50 [ 2984.663759] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b958781c800.ost_server_uuid in FULL state after 0 sec [ 2988.523082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.528688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2988.533059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2988.533068] Lustre: Skipped 1 previous similar message [ 2993.081607] Lustre: server umount lustre-OST0000 complete [ 2998.760970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.764907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.771060] Lustre: Skipped 2 previous similar messages [ 2998.784017] Lustre: Skipped 2 previous similar messages [ 3002.241511] Lustre: server umount lustre-MDT0000 complete [ 3003.873720] LustreError: 169042:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.890910] LustreError: 169042:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3004.937132] LustreError: 169021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235698 with bad export cookie 9838080978191606441 [ 3004.943523] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3004.946143] LustreError: 169021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3005.157533] Lustre: server umount lustre-MDT0001 complete [ 3012.869824] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 3015.263551] Key type lgssc unregistered [ 3015.497312] LNet: 173690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3015.508164] LNetError: 173690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3015.518032] LNet: Removed LNI 192.168.201.110@tcp [ 3016.164141] Key type .llcrypt unregistered [ 3016.166934] Key type ._llcrypt unregistered [ 3038.330392] Key type ._llcrypt registered [ 3038.332322] Key type .llcrypt registered [ 3038.408081] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3038.998959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3039.106033] alg: No test for adler32 (adler32-zlib) [ 3040.062413] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 3040.218429] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 3041.872316] Key type lgssc registered [ 3042.646819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.546405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.563958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.953385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.551962] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3058.867549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.113508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.590828] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3063.591962] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3068.696893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3068.940497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.951963] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3075.113459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 3079.801876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.978379] Lustre: lustre-OST0001: new disk, initializing [ 3079.983355] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3080.070817] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.559312] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3086.408848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3086.429980] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3086.460135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3092.911193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3100.031936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3117.026551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.028877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.041971] Lustre: Skipped 3 previous similar messages [ 3117.047444] Lustre: Skipped 3 previous similar messages [ 3118.982722] Lustre: server umount lustre-MDT0000 complete [ 3121.582780] LustreError: 175359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235815 with bad export cookie 14317073125885407318 [ 3121.586419] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3121.589832] LustreError: 175359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3121.796441] Lustre: server umount lustre-MDT0001 complete [ 3135.025450] Lustre: server umount lustre-OST0000 complete [ 3138.017312] Lustre: 174645:0:(client.c:2516:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773235816/real 1773235816] req@ffff9e65d3379180 x1859372433102976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773235832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3138.036697] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.250217] Lustre: server umount lustre-OST0001 complete [ 3152.006938] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3157.890268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.211966] LustreError: 180710:0:(ldlm_lib.c:1178:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3158.221976] LustreError: 180710:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3158.266390] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.388319] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3163.619778] LustreError: 180711:0:(ldlm_lib.c:1178:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.832843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.062930] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3171.277739] Lustre: 181817:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3176.565651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3176.882607] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.890774] Lustre: Skipped 1 previous similar message [ 3181.222983] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3183.016939] LustreError: 182172:0:(ldlm_lib.c:1178:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.022042] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:323 to 0x280000401:353) [ 3187.304610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3188.680383] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3191.662118] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3197.212302] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3199.427455] Lustre: 183659:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3212.262400] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3212.263662] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.280043] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.283720] Lustre: Skipped 2 previous similar messages [ 3212.287375] Lustre: Skipped 1 previous similar message [ 3217.378745] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.386988] Lustre: Skipped 5 previous similar messages [ 3218.322404] Lustre: server umount mylustre-MDT0000 complete [ 3220.628628] LustreError: 180687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235914 with bad export cookie 14317073125885410125 [ 3220.635323] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.641388] LustreError: 180687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3220.847474] Lustre: server umount mylustre-MDT0001 complete [ 3233.494618] Lustre: server umount mylustre-OST0000 complete [ 3246.024834] Lustre: server umount mylustre-OST0001 complete [ 3259.262130] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3264.621741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.937830] LustreError: 185948:0:(ldlm_lib.c:1178:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.993249] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.997275] Lustre: Skipped 1 previous similar message [ 3267.048753] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3270.113888] LustreError: 185949:0:(ldlm_lib.c:1178:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.171939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.862107] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3276.761180] Lustre: 187056:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3281.576473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.861081] LustreError: 187410:0:(ldlm_lib.c:1178:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.707587] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3289.071349] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3291.259512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.320910] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 3294.971804] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3300.355414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3302.730417] Lustre: 188900:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3312.103186] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.107294] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.112659] Lustre: Skipped 2 previous similar messages [ 3317.219742] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3317.227902] Lustre: Skipped 6 previous similar messages [ 3317.315662] Lustre: server umount tfs-MDT0000 complete [ 3319.249323] LustreError: 185924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236013 with bad export cookie 14317073125885412015 [ 3319.255664] LustreError: 185924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3319.255862] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.445925] Lustre: server umount tfs-MDT0001 complete [ 3331.606996] Lustre: server umount tfs-OST0000 complete [ 3343.907109] Lustre: server umount tfs-OST0001 complete [ 3356.425313] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3361.962859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.253946] LustreError: 191130:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.263093] LustreError: 191130:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3362.314858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.317893] Lustre: Skipped 3 previous similar messages [ 3365.185036] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3370.224232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.247463] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3375.232443] Lustre: 192238:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3379.382572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.348299] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3387.886166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3388.341551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.542079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 3391.793310] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3396.683977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3398.952562] Lustre: 194080:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3408.381520] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 09:35:01 (1773236101) [ 3424.229113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.238639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.247082] Lustre: Skipped 5 previous similar messages [ 3424.250042] Lustre: Skipped 2 previous similar messages [ 3428.128415] Lustre: server umount lustre-MDT0000 complete [ 3429.346296] LustreError: 191131:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.357149] LustreError: 191131:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 3430.493626] LustreError: 191108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236124 with bad export cookie 14317073125885413681 [ 3430.496184] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.512832] LustreError: 191108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3434.467632] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.778957] Lustre: server umount lustre-MDT0001 complete [ 3445.081179] Lustre: server umount lustre-OST0000 complete [ 3453.130386] Lustre: server umount lustre-OST0001 complete [ 3456.530332] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3463.496427] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3486.837682] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3486.837682] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3486.837682] [ 3486.859936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3487.033449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3487.057066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3487.110934] Lustre: lustre-MDT0000: new disk, initializing [ 3487.188600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.195026] Lustre: Skipped 3 previous similar messages [ 3487.212372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3489.548769] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3496.117602] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 3496.117602] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3496.117602] [ 3496.136255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3496.197705] Lustre: 197231:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3496.244678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3496.249291] Lustre: Skipped 1 previous similar message [ 3496.299596] Lustre: lustre-MDT0001: new disk, initializing [ 3496.371529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3496.378839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3498.434874] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3505.696940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.946124] Lustre: lustre-OST0000: new disk, initializing [ 3505.950023] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3507.284958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3507.289836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3507.362516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3509.601952] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3516.030542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.130326] Lustre: lustre-OST0001: new disk, initializing [ 3516.134248] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3517.357219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3517.363241] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3517.437141] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3519.464900] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3526.295994] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 09:36:59 (1773236219) [ 3532.594868] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 09:37:05 (1773236225) [ 3548.134336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.138355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.141264] Lustre: Skipped 2 previous similar messages [ 3548.144991] Lustre: Skipped 5 previous similar messages [ 3559.392188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3559.461487] Lustre: server umount lustre-OST0000 complete [ 3562.977463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3563.493534] LustreError: 198909:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.507306] LustreError: 198909:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 3567.886778] Lustre: server umount lustre-MDT0000 complete [ 3569.993742] LustreError: 197223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236264 with bad export cookie 14317073125885415235 [ 3569.996667] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3570.001145] LustreError: 197223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3576.228741] Lustre: server umount lustre-MDT0001 complete [ 3582.895660] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 3596.274808] Lustre: server umount lustre-OST0001 complete [ 3600.312269] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3610.801383] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3633.122599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.388339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.431704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.503670] Lustre: lustre-MDT0000: new disk, initializing [ 3633.579623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.583668] Lustre: Skipped 3 previous similar messages [ 3633.600194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3635.973329] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3642.527021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.622705] Lustre: 204097:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3642.828656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3645.087937] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3650.000443] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3652.351433] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3656.477939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3656.628778] Lustre: lustre-OST0000: new disk, initializing [ 3656.631152] Lustre: Skipped 1 previous similar message [ 3656.635263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3656.641403] Lustre: Skipped 2 previous similar messages [ 3658.599330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3658.601936] Lustre: Skipped 1 previous similar message [ 3658.605322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3658.680671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3660.426983] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3665.311575] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3674.086774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.086709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.103911] Lustre: Skipped 4 previous similar messages [ 3674.109636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.116080] Lustre: Skipped 11 previous similar messages [ 3675.204761] Lustre: server umount lustre-OST0000 complete [ 3678.690065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3678.696665] LustreError: Skipped 1 previous similar message [ 3683.404677] Lustre: server umount lustre-MDT0000 complete [ 3685.454958] LustreError: 204087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236379 with bad export cookie 14317073125885416705 [ 3685.460085] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.465146] LustreError: 204087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3685.695307] Lustre: server umount lustre-MDT0001 complete [ 3691.841273] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 3693.615155] Key type lgssc unregistered [ 3693.778647] LNet: 207467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3693.782734] LNetError: 207467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3693.796737] LNet: Removed LNI 192.168.201.110@tcp [ 3694.269513] Key type .llcrypt unregistered [ 3694.272330] Key type ._llcrypt unregistered [ 3709.606856] Key type ._llcrypt registered [ 3709.611058] Key type .llcrypt registered [ 3709.682185] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3710.134974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3710.194053] alg: No test for adler32 (adler32-zlib) [ 3711.097763] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 3711.230772] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 3712.840178] Key type lgssc registered [ 3713.436214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3718.217831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3718.228784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.535212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.789599] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3726.770866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.038770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.138309] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3732.314867] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3734.549798] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3738.316822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.577596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.063156] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3744.752347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3745.173347] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3745.856614] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3749.859250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3749.863946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3749.868542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.931591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.017913] Lustre: server umount lustre-OST0000 complete [ 3758.050303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3758.051139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.052791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.052801] Lustre: Skipped 1 previous similar message [ 3758.065891] Lustre: Skipped 1 previous similar message [ 3761.321270] Lustre: server umount lustre-MDT0000 complete [ 3763.169147] LustreError: 209603:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.178345] LustreError: 209603:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3763.244547] LustreError: 211084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236457 with bad export cookie 17459511084850470437 [ 3763.250152] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3763.251126] LustreError: 211084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3763.429464] Lustre: server umount lustre-MDT0001 complete [ 3768.744040] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 3770.312890] Key type lgssc unregistered [ 3770.488449] LNet: 212121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3770.494228] LNetError: 212121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3770.515729] LNet: Removed LNI 192.168.201.110@tcp [ 3770.986232] Key type .llcrypt unregistered [ 3770.988222] Key type ._llcrypt unregistered [ 3779.986269] Key type ._llcrypt registered [ 3779.988603] Key type .llcrypt registered [ 3780.048106] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3780.627881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3780.649061] alg: No test for adler32 (adler32-zlib) [ 3781.597345] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 3781.734725] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 3783.360263] Key type lgssc registered [ 3783.993410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3789.002504] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3789.019339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3804.512669] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3809.632423] LustreError: 213275:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.110@tcp: failed processing log, type 4: rc = -110 [ 3836.256257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.119378] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 3842.021970] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3843.191885] Lustre: server umount lustre-OST0000 complete [ 3851.142328] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 3852.577375] Key type lgssc unregistered [ 3852.753167] LNet: 214776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3852.764698] LNetError: 214776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3853.799175] LNet: Removed LNI 192.168.201.110@tcp [ 3854.230285] Key type .llcrypt unregistered [ 3854.232915] Key type ._llcrypt unregistered [ 3874.787158] Key type ._llcrypt registered [ 3874.789744] Key type .llcrypt registered [ 3874.850848] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3875.304657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3875.348040] alg: No test for adler32 (adler32-zlib) [ 3876.255307] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 3876.378146] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 3877.985848] Key type lgssc registered [ 3878.569953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3885.100488] loop0: detected capacity change from 0 to 204800000 [ 3885.195475] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3885.201379] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3885.392993] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3885.928659] loop0: detected capacity change from 0 to 204800000 [ 3886.074829] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3886.081583] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3886.293939] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3886.846369] loop0: detected capacity change from 0 to 204800000 [ 3886.938777] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3886.954816] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3887.027456] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3887.565366] loop0: detected capacity change from 0 to 204800000 [ 3887.721744] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3888.336136] loop0: detected capacity change from 0 to 204800000 [ 3888.359289] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3890.722054] loop0: detected capacity change from 0 to 204800000 [ 3890.738912] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3893.108923] loop0: detected capacity change from 0 to 204800000 [ 3893.127341] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3895.525775] loop0: detected capacity change from 0 to 204800000 [ 3895.543546] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3897.622410] loop0: detected capacity change from 0 to 204800000 [ 3897.671341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3897.685379] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.759645] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3897.765324] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3897.771563] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3897.778802] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3897.785272] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3897.792369] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3897.798874] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3897.806847] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3897.813943] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3897.821335] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3897.827400] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3897.834309] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3897.842683] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3897.848221] Lustre: 218153:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3901.261214] Lustre: server umount MGS complete [ 3901.716586] loop0: detected capacity change from 0 to 204800000 [ 3901.757311] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3902.013095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.035211] Lustre: lustre-MDT0000: Aborting client recovery [ 3902.458538] loop1: detected capacity change from 0 to 204800000 [ 3902.509497] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3902.553669] Lustre: 218584:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3902.560069] Lustre: 218584:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3902.568712] Lustre: 218584:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3902.576746] Lustre: 218584:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3902.583726] Lustre: 218584:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3902.591451] Lustre: 218584:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3902.601110] Lustre: 218584:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3902.613272] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3902.718193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3902.760514] Lustre: lustre-MDT0001: Aborting client recovery [ 3903.212913] loop2: detected capacity change from 0 to 204800000 [ 3903.276067] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.315541] Lustre: 218667:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3903.322795] Lustre: 218667:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3903.329291] Lustre: 218667:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3903.337387] Lustre: 218667:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3903.344314] Lustre: 218667:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3903.351968] Lustre: 218667:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3903.359578] Lustre: 218667:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3903.428727] Lustre: Found index 0 for lustre-OST0000, updating log [ 3903.487610] Lustre: lustre-OST0000: Aborting recovery [ 3903.905330] loop3: detected capacity change from 0 to 204800000 [ 3903.940634] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.999241] Lustre: 218751:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3904.006713] Lustre: 218751:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3904.015426] Lustre: 218751:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3904.025640] Lustre: 218751:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3904.034419] Lustre: 218751:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3904.041906] Lustre: 218751:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3904.047941] Lustre: 218751:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3904.109781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3904.114340] Lustre: Skipped 1 previous similar message [ 3904.126227] Lustre: lustre-OST0001: Aborting recovery [ 3909.624848] LustreError: 218528:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 3912.677782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3912.680172] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3912.680290] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3912.689308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3912.693801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3912.720205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3915.687429] LustreError: 218495:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 3316us: rc = -2 [ 3928.036145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.039180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.050577] Lustre: Skipped 2 previous similar messages [ 3928.054682] Lustre: Skipped 1 previous similar message [ 3933.156943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3933.162519] Lustre: Skipped 3 previous similar messages [ 3933.733534] Lustre: server umount lustre-MDT0000 complete [ 3934.276790] LustreError: 218592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236628 with bad export cookie 14423167597240253766 [ 3934.278746] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3934.289210] LustreError: 218592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3938.273660] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.274050] LustreError: 218493:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.274486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3938.280228] Lustre: Skipped 2 previous similar messages [ 3938.296870] LustreError: 218493:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3939.296729] LustreError: 218693:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.304407] LustreError: 218693:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3940.565022] Lustre: server umount lustre-MDT0001 complete [ 3947.091323] Lustre: server umount lustre-OST0000 complete [ 3953.664695] Lustre: server umount lustre-OST0001 complete [ 3957.493664] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 09:44:10 (1773236650) [ 3970.430500] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 3975.214101] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 3993.278913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.390253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3993.408515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3993.460233] Lustre: lustre-MDT0000: new disk, initializing [ 3993.494215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.504437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3995.321905] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4000.535501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.593700] Lustre: 222793:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4000.626937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4000.630042] Lustre: Skipped 1 previous similar message [ 4000.680541] Lustre: lustre-MDT0001: new disk, initializing [ 4000.722673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4000.738462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4000.746385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4002.550088] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4006.941462] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4008.792893] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4011.992096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.122344] Lustre: lustre-OST0000: new disk, initializing [ 4012.125714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4012.164940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4014.743786] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4017.670471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4017.675177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4017.733351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4018.790974] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4021.777207] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4027.593855] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4034.000788] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4079.080148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4079.080926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.084394] LustreError: Skipped 1 previous similar message [ 4079.090313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.093737] Lustre: Skipped 4 previous similar messages [ 4083.502264] Lustre: server umount lustre-OST0000 complete [ 4087.776859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4087.780926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.787549] Lustre: Skipped 1 previous similar message [ 4087.791175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.443603] Lustre: server umount lustre-MDT0000 complete [ 4093.170497] LustreError: 223571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236787 with bad export cookie 14423167597240256475 [ 4093.175523] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.178804] LustreError: 223571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4093.329192] Lustre: server umount lustre-MDT0001 complete [ 4097.423669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.374346] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4101.920316] Lustre: server umount MGS complete [ 4107.161854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.383982] LustreError: 228312:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.394740] LustreError: 228325:0:(qmt_pool.c:1614:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4107.422635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.428554] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4112.866253] LustreError: 228313:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.678282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.771410] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4118.465245] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4120.189700] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4123.629059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.673447] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4129.478361] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4158.947700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4158.951424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.954441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4158.958437] Lustre: Skipped 2 previous similar messages [ 4158.961804] Lustre: Skipped 2 previous similar messages [ 4160.061128] Lustre: server umount lustre-OST0000 complete [ 4176.352165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4176.427671] Lustre: server umount lustre-MDT0000 complete [ 4178.050548] LustreError: 228294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236872 with bad export cookie 14423167597240258302 [ 4178.054220] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4178.056919] LustreError: 228294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4178.208517] Lustre: server umount lustre-MDT0001 complete [ 4183.151298] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4184.415864] Key type lgssc unregistered [ 4184.561680] LNet: 231905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4184.564813] LNetError: 231905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4184.575481] LNet: Removed LNI 192.168.201.110@tcp [ 4184.943842] Key type .llcrypt unregistered [ 4184.945546] Key type ._llcrypt unregistered [ 4202.304333] Key type ._llcrypt registered [ 4202.306774] Key type .llcrypt registered [ 4202.367295] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4210.268281] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4210.833074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4210.848737] alg: No test for adler32 (adler32-zlib) [ 4211.730156] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4211.845598] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 4213.440277] Key type lgssc registered [ 4213.937286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4230.580461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4230.589945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.703730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4231.746161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4231.797032] Lustre: lustre-MDT0000: new disk, initializing [ 4231.834634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.844942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4233.580892] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4238.710868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.773021] Lustre: 235798:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4238.810214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4238.813043] Lustre: Skipped 1 previous similar message [ 4238.866578] Lustre: lustre-MDT0001: new disk, initializing [ 4238.915837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4238.935939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4238.942338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4240.775143] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4244.647759] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4246.313437] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4249.444782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.569347] Lustre: lustre-OST0000: new disk, initializing [ 4249.572080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4249.609297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.172615] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4255.845265] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4259.845763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4259.852056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4259.904307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4265.061649] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4269.327233] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4274.306354] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4316.130724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4316.132913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.133625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4316.141436] LustreError: Skipped 1 previous similar message [ 4316.148611] Lustre: Skipped 1 previous similar message [ 4317.668125] Lustre: server umount lustre-OST0000 complete [ 4320.736859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4320.740272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.745850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4320.748919] Lustre: Skipped 1 previous similar message [ 4325.350538] Lustre: server umount lustre-MDT0000 complete [ 4326.856439] LustreError: 235792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237020 with bad export cookie 4768699427115692615 [ 4326.859618] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.860694] LustreError: 235792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.991903] Lustre: server umount lustre-MDT0001 complete [ 4330.869702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.489343] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4334.573706] Lustre: server umount MGS complete [ 4339.174097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.425530] LustreError: 241217:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.435941] LustreError: 241217:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4339.443175] LustreError: 241230:0:(qmt_pool.c:1614:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4339.483588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.387815] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4344.801997] LustreError: 241218:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.212502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.378363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4347.046360] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4349.326560] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4350.939690] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4353.895178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4354.077558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4356.571078] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4359.011263] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4384.740290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.741335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.746497] Lustre: Skipped 2 previous similar messages [ 4384.747271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.749666] Lustre: Skipped 3 previous similar messages [ 4387.824044] Lustre: server umount lustre-OST0000 complete [ 4389.857874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.860503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.865458] Lustre: Skipped 1 previous similar message [ 4389.871674] Lustre: Skipped 1 previous similar message [ 4394.979174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4394.982375] Lustre: Skipped 1 previous similar message [ 4395.750609] Lustre: server umount lustre-MDT0000 complete [ 4397.419911] LustreError: 241199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237091 with bad export cookie 4768699427115694330 [ 4397.424654] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4397.424962] LustreError: 241199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4397.564113] Lustre: server umount lustre-MDT0001 complete [ 4402.757422] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4404.118747] Key type lgssc unregistered [ 4404.297366] LNet: 244806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4404.301980] LNetError: 244806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4404.317396] LNet: Removed LNI 192.168.201.110@tcp [ 4404.730594] Key type .llcrypt unregistered [ 4404.732170] Key type ._llcrypt unregistered [ 4414.120376] Key type ._llcrypt registered [ 4414.122590] Key type .llcrypt registered [ 4414.171715] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4421.254938] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4421.667528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.713027] alg: No test for adler32 (adler32-zlib) [ 4422.601136] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4422.697392] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 4424.288216] Key type lgssc registered [ 4424.792601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4435.305582] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4441.672837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4441.682670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.805183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4442.821452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4442.868991] Lustre: lustre-MDT0000: new disk, initializing [ 4442.900136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.908888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4444.670217] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4449.471588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4449.607414] Lustre: lustre-OST0000: new disk, initializing [ 4449.610317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4449.614221] Lustre: Skipped 1 previous similar message [ 4449.648474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.238145] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4458.506774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4458.511432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4458.521962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4678.625090] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.628948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.195393] Lustre: server umount lustre-MDT0000 complete [ 4681.661375] LustreError: 247964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237375 with bad export cookie 11861308495902252482 [ 4681.663345] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4681.666048] LustreError: 247964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4681.740451] Lustre: server umount lustre-OST0000 complete [ 4691.235752] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 4694.823216] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4708.308846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.402085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4708.414915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4708.454516] Lustre: lustre-MDT0000: new disk, initializing [ 4708.481186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.489905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4709.976805] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4714.609229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.667517] Lustre: 252945:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4714.696342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4714.699028] Lustre: Skipped 1 previous similar message [ 4714.743700] Lustre: lustre-MDT0001: new disk, initializing [ 4714.784758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4714.801884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4714.808190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4716.235026] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4719.337190] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4720.541938] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4722.781318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4722.894764] Lustre: lustre-OST0000: new disk, initializing [ 4722.898150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4722.932300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4724.790639] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4725.693297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4725.697676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4725.733643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4728.112026] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4729.635712] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4729.703404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4731.231218] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4731.325357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4735.973204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4735.974029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.975168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4735.975677] LustreError: Skipped 1 previous similar message [ 4735.979741] Lustre: Skipped 1 previous similar message [ 4738.141108] Lustre: server umount lustre-OST0000 complete [ 4740.576862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4740.579591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.584630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4740.586743] Lustre: Skipped 1 previous similar message [ 4745.692560] Lustre: server umount lustre-MDT0000 complete [ 4747.120569] LustreError: 252938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237441 with bad export cookie 11861308495905193280 [ 4747.125629] LustreError: 252938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4747.125966] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4747.259125] Lustre: server umount lustre-MDT0001 complete [ 4748.727592] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 4749.345352] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 09:57:22 (1773237442) [ 4751.763343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.927901] LustreError: 256849:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.936600] LustreError: 256849:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4751.957784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.453086] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4756.766906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4758.427879] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4760.562837] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4761.914017] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4764.458311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4766.559431] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4768.651949] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4771.374106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.451510] Lustre: lustre-OST0001: new disk, initializing [ 4771.454463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4771.487932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4771.490432] Lustre: Skipped 2 previous similar messages [ 4771.919634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4771.923477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4771.944364] LustreError: 259416:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4772.964558] LustreError: 259426:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4772.969076] LustreError: 259426:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 4773.640511] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4773.987825] LustreError: 259416:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4773.991596] LustreError: 259416:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4776.042229] LustreError: 259426:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4776.048576] LustreError: 259426:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 4777.603647] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4777.693603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4778.994774] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b9589987800.ost_server_uuid 50 [ 4779.617772] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b9589987800.ost_server_uuid in FULL state after 0 sec [ 4780.134053] LustreError: 259426:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4780.137695] LustreError: 259426:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 4780.914216] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b9589987800.ost_server_uuid 50 [ 4781.494433] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b9589987800.ost_server_uuid in FULL state after 0 sec [ 4782.186142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4796.842447] Lustre: lustre-OST0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 4796.846746] Lustre: Skipped 2 previous similar messages [ 4796.901933] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.908143] Lustre: Skipped 2 previous similar messages [ 4799.425434] Lustre: server umount lustre-OST0001 complete [ 4802.019222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.020801] LustreError: 258443:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4802.021944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4802.021951] Lustre: Skipped 2 previous similar messages [ 4802.022779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4802.023874] Lustre: Skipped 1 previous similar message [ 4802.039577] LustreError: 258443:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4807.141346] LustreError: 259746:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4807.149223] LustreError: 259746:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4807.314264] Lustre: server umount lustre-OST0000 complete [ 4812.257721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.258410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4812.261103] Lustre: Skipped 1 previous similar message [ 4812.264260] Lustre: Skipped 4 previous similar messages [ 4823.008061] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4823.073356] Lustre: server umount lustre-MDT0000 complete [ 4824.227976] LustreError: 256831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237518 with bad export cookie 11861308495905194148 [ 4824.230558] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4824.231879] LustreError: 256831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4824.331573] Lustre: server umount lustre-MDT0001 complete [ 4827.942523] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4829.022994] Key type lgssc unregistered [ 4829.159347] LNet: 261788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4829.161808] LNetError: 261788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4829.170350] LNet: Removed LNI 192.168.201.110@tcp [ 4829.476336] Key type .llcrypt unregistered [ 4829.477774] Key type ._llcrypt unregistered [ 4838.609650] Key type ._llcrypt registered [ 4838.611054] Key type .llcrypt registered [ 4838.662307] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4839.025549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4839.040885] alg: No test for adler32 (adler32-zlib) [ 4839.897714] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4839.988848] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 4841.568142] Key type lgssc registered [ 4841.973090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4844.646730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4844.653188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.803222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4847.051812] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4848.801307] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4851.082464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.203491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4852.384083] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4855.269683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4855.330449] LustreError: 264714:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.336679] LustreError: 264714:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4855.384703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.308872] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4859.043496] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4860.387224] LustreError: 264715:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4860.393839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4861.360179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4861.412878] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 4861.417345] Lustre: Skipped 1 previous similar message [ 4861.455549] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4862.614704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4863.648431] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4865.379390] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4867.858076] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4867.936138] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4944.865176] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4944.866070] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.867490] LustreError: Skipped 1 previous similar message [ 4944.873162] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4944.875477] Lustre: Skipped 1 previous similar message [ 4948.450584] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4948.453040] Lustre: Skipped 1 previous similar message [ 4949.565951] Lustre: server umount lustre-OST0001 complete [ 4952.545213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4952.549353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.553250] Lustre: Skipped 1 previous similar message [ 4952.555646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4953.569662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4953.571697] LustreError: 264715:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.572152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.580361] LustreError: 264715:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4957.260307] Lustre: server umount lustre-OST0000 complete [ 4958.691462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.691530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4958.692372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4958.692379] Lustre: Skipped 2 previous similar messages [ 4958.695463] Lustre: Skipped 1 previous similar message [ 4964.700299] Lustre: server umount lustre-MDT0000 complete [ 4965.952260] LustreError: 266476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237659 with bad export cookie 16987497273400186137 [ 4965.955888] LustreError: 266476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4965.956134] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4966.074523] Lustre: server umount lustre-MDT0001 complete [ 4969.614766] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 4970.621714] Key type lgssc unregistered [ 4970.754492] LNet: 267725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4970.757347] LNetError: 267725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4970.766324] LNet: Removed LNI 192.168.201.110@tcp [ 4971.091371] Key type .llcrypt unregistered [ 4971.092962] Key type ._llcrypt unregistered [ 4987.969997] Key type ._llcrypt registered [ 4987.971198] Key type .llcrypt registered [ 4988.003791] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 4988.405398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4988.412635] alg: No test for adler32 (adler32-zlib) [ 4989.248321] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 4989.322063] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 4990.896161] Key type lgssc registered [ 4991.227433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4994.070353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4994.077065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.233438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.606613] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 4999.451363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.586834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5000.893287] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5001.869836] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5004.081797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5004.210425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.220206] LustreError: 271429:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.224356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5005.226583] LustreError: 271429:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5006.161987] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5008.977797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5009.052251] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5010.091912] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 5010.887275] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5015.107360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5017.746178] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5030.369031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5030.371113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.374800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5030.884033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.887031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5030.888718] Lustre: Skipped 2 previous similar messages [ 5030.891202] Lustre: Skipped 2 previous similar messages [ 5032.105366] Lustre: server umount lustre-MDT0000 complete [ 5033.362322] LustreError: 271078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237727 with bad export cookie 2028763164657638017 [ 5033.365955] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5033.366742] LustreError: 271078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5033.490312] Lustre: server umount lustre-MDT0001 complete [ 5044.228389] Lustre: server umount lustre-OST0000 complete [ 5055.775695] Lustre: server umount lustre-OST0001 complete [ 5061.198306] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 5064.612734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5064.774167] LustreError: 275157:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.796812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.193124] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5069.073942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.427083] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5071.349153] Lustre: 276264:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5073.401576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.512428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.515078] Lustre: Skipped 1 previous similar message [ 5074.532047] LustreError: 276617:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.535066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5075.282095] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5078.001033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5079.146271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 5079.889791] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5082.925129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5084.020384] Lustre: 278107:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5094.885544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.886574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.890730] Lustre: Skipped 2 previous similar messages [ 5094.892551] Lustre: Skipped 1 previous similar message [ 5094.897027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5096.156976] Lustre: server umount lustre-MDT0000 complete [ 5097.420172] LustreError: 275138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237791 with bad export cookie 2028763164657639704 [ 5097.423148] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.423665] LustreError: 275138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5097.547896] Lustre: server umount lustre-MDT0001 complete [ 5108.236861] Lustre: server umount lustre-OST0000 complete [ 5119.767458] Lustre: server umount lustre-OST0001 complete [ 5121.066426] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 5121.655690] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 5122.285452] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 10:03:35 (1773237815) [ 5129.645450] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 5132.655885] loop0: detected capacity change from 0 to 1099511627776 [ 5132.839127] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 5132.842106] *** VALIDATE xfs *** [ 5132.847381] XFS (loop0): Mounting V5 Filesystem [ 5132.857613] XFS (loop0): Ending clean mount [ 5133.285362] loop1: detected capacity change from 0 to 18691697672192 [ 5154.725475] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 5158.250024] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 5170.409751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.505297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5170.516659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5170.552529] Lustre: lustre-MDT0000: new disk, initializing [ 5170.579135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5170.580978] Lustre: Skipped 1 previous similar message [ 5170.586899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5171.845926] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5176.061228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.102455] Lustre: 283171:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5176.127462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5176.129291] Lustre: Skipped 1 previous similar message [ 5176.168102] Lustre: lustre-MDT0001: new disk, initializing [ 5176.217930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5176.221770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5177.506303] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5180.634955] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5181.921412] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5184.197589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5184.309030] Lustre: lustre-OST0000: new disk, initializing [ 5184.311080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5185.466130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5185.469486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5185.503726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5186.339777] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5189.521295] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5190.907213] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5190.987353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5192.309230] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5192.379427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5195.091233] Lustre: server umount lustre-OST0000 complete [ 5196.769108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5196.771844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.775802] Lustre: Skipped 1 previous similar message [ 5196.777939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5196.779987] Lustre: Skipped 2 previous similar messages [ 5200.865561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5200.866475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5206.014083] Lustre: server umount lustre-MDT0000 complete [ 5207.307222] LustreError: 283950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237901 with bad export cookie 2028763164657641258 [ 5207.310516] LustreError: 283950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5207.310792] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.820258] Lustre: server umount lustre-MDT0001 complete [ 5220.575872] XFS (loop0): Unmounting Filesystem [ 5222.134770] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 10:05:15 (1773237915) [ 5224.395256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.532264] LustreError: 287322:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.553200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.555818] Lustre: Skipped 2 previous similar messages [ 5225.697385] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5228.547818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.033992] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5231.849486] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5233.092242] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5235.432096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.548240] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5239.442224] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5259.232962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5259.235764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.240974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5259.242716] Lustre: Skipped 3 previous similar messages [ 5264.274713] Lustre: server umount lustre-OST0000 complete [ 5271.704395] Lustre: server umount lustre-MDT0000 complete [ 5272.966775] LustreError: 287303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237967 with bad export cookie 2028763164657642140 [ 5272.970079] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.971770] LustreError: 287303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5273.108463] Lustre: server umount lustre-MDT0001 complete [ 5277.036213] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 5278.191511] Key type lgssc unregistered [ 5278.336376] LNet: 290616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5278.339145] LNetError: 290616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5278.345348] LNet: Removed LNI 192.168.201.110@tcp [ 5278.684110] Key type .llcrypt unregistered [ 5278.685161] Key type ._llcrypt unregistered [ 5289.289576] Key type ._llcrypt registered [ 5289.291324] Key type .llcrypt registered [ 5289.329913] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 5289.756590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5289.805990] alg: No test for adler32 (adler32-zlib) [ 5290.666952] Lustre: Lustre: Build Version: 2.17.50_236_gaab9274 [ 5290.763403] LNet: Added LNI 192.168.201.110@tcp [8/256/0/180] [ 5292.352136] Key type lgssc registered [ 5292.734665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5295.401859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5295.408390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.570348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5297.919302] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5300.749832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.868711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5302.200451] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5304.070945] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5305.270185] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5307.517761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5307.628987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5309.520913] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5311.361558] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5326.816973] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5326.819745] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.824079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5327.331295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5327.333787] Lustre: Skipped 1 previous similar message [ 5327.334657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.384126] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 5332.387027] Lustre: Skipped 1 previous similar message [ 5337.504078] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.10@tcp (stopping) [ 5337.507939] Lustre: Skipped 2 previous similar messages [ 5339.104140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5339.162478] Lustre: server umount lustre-MDT0001 complete [ 5342.623580] LustreError: 292139:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5342.629683] LustreError: 292139:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5342.686326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.724321] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5342.726634] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5342.740059] Lustre: 292132:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5342.850481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5344.166670] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5348.326565] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5351.398780] LustreError: 295625:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5351.403191] LustreError: 295625:0:(tgt_mount.c:667:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5351.407094] LustreError: 292135:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5351.428344] LustreError: 295628:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.110@tcp: cfg command failed: rc = -17 [ 5351.433563] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5351.433563] [ 5648.353702] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.359616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5648.361391] Lustre: Skipped 3 previous similar messages [ 5658.593567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5658.596127] Lustre: Skipped 5 previous similar messages [ 5660.128088] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5660.186837] Lustre: server umount lustre-MDT0001 complete [ 5663.662762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.701579] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5663.703326] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5663.714924] LustreError: 292137:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.720666] LustreError: 292137:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 5663.721795] Lustre: 292133:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5663.842960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5665.096983] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5669.351702] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5673.960910] LustreError: 308183:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5673.963876] LustreError: 308183:0:(tgt_mount.c:667:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5673.966309] LustreError: 292135:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5673.968793] LustreError: 292135:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5673.987929] LustreError: 308186:0:(obd_config.c:797:class_setup()) Device 8 already setup (type osp) [ 5673.990248] LustreError: 308186:0:(obd_config.c:902:class_detach()) OBD device 8 still set up [ 5673.992837] LustreError: 308186:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.110@tcp: cfg command failed: rc = -17 [ 5673.996667] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.110@tcp [ 5673.996667] [ 5971.429027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.429219] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5971.432810] Lustre: Skipped 1 previous similar message [ 5971.436143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5971.438563] Lustre: Skipped 2 previous similar messages [ 5973.593424] Lustre: server umount lustre-MDT0001 complete [ 5976.548522] LustreError: 294295:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.552960] LustreError: 294295:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5977.069431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.102683] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5977.104782] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5977.119321] Lustre: 292130:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5977.249119] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5978.596738] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 5982.695923] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5985.765877] LustreError: 320740:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5985.768623] LustreError: 320740:0:(tgt_mount.c:667:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5985.771246] LustreError: 292135:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5985.774577] LustreError: 292135:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5985.790983] LustreError: 320743:0:(lod_lov.c:212:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 5985.796043] LustreError: 320743:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.110@tcp: cfg command failed: rc = -114 [ 5985.798980] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 5985.798980] [ 6295.013662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6295.013682] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.014615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.021368] Lustre: Skipped 3 previous similar messages [ 6298.838267] Lustre: server umount lustre-MDT0000 complete [ 6300.029410] LustreError: 292167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238994 with bad export cookie 8145904307736858093 [ 6300.032958] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.034525] LustreError: 292167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6300.128985] LustreError: 293485:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.249061] LustreError: 294295:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.221616] Lustre: server umount lustre-MDT0001 complete [ 6307.505366] Lustre: server umount lustre-OST0000 complete [ 6310.410936] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6313.575523] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6325.344236] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 10:23:38 (1773239018) [ 6327.582102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.678318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6327.689923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6327.727862] Lustre: lustre-MDT0000: new disk, initializing [ 6327.753556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.761100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6329.053507] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6333.139511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.179114] Lustre: 335268:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6333.203656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6333.205427] Lustre: Skipped 1 previous similar message [ 6333.240806] Lustre: lustre-MDT0001: new disk, initializing [ 6333.273312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6333.284725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6333.287947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6334.539468] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6337.514240] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6338.660128] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6340.877338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6340.990533] Lustre: lustre-OST0000: new disk, initializing [ 6340.992381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6341.021232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.863172] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_default_debug -1 all [ 6345.014883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6345.019461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6345.050348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6345.936822] Lustre: DEBUG MARKER: oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6348.768884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6348.771355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.774893] Lustre: Skipped 3 previous similar messages [ 6354.587089] Lustre: server umount lustre-MDT0000 complete [ 6355.425129] LustreError: 335280:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.429322] LustreError: 335280:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6355.791123] LustreError: 335260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239049 with bad export cookie 8145904307736860032 [ 6355.793780] LustreError: 336059:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 6355.794758] LustreError: MGC192.168.201.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.796322] LustreError: 335260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6355.912268] Lustre: server umount lustre-MDT0001 complete [ 6367.455786] Lustre: server umount lustre-OST0000 complete [ 6376.562461] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing set_hostid [ 6379.768745] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing load_modules_local [ 6390.254911] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6243 sec ========== 10:24:43 (1773239083) [ 6390.846244] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:24:44 (1773239084) === [ 6391.989023] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:24:45 (1773239085) === [ 6404.210621] Lustre: DEBUG MARKER: oleg110-server.virtnet: executing unload_modules_local [ 6405.142799] Key type lgssc unregistered [ 6405.268342] LNet: 343210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6405.270265] LNetError: 343210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6405.277251] LNet: Removed LNI 192.168.201.110@tcp [ 6405.556079] Key type .llcrypt unregistered [ 6405.557466] Key type ._llcrypt unregistered