[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488485849 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002401] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010031] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012052] SELinux: Initializing. [ 0.013100] *** VALIDATE selinux *** [ 0.021470] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025814] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] *** VALIDATE tmpfs *** [ 0.030510] *** VALIDATE proc *** [ 0.031289] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.034046] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036060] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.043788] debug: unmapping init [mem 0xffffffff8ec59000-0xffffffff8ec60fff] [ 0.045168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046724] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053018] ... event mask: 000000070000000f [ 0.054328] rcu: Hierarchical SRCU implementation. [ 0.056521] smp: Bringing up secondary CPUs ... [ 0.057622] x86: Booting SMP configuration: [ 0.058029] .... node #0, CPUs: #1 #2 #3 [ 0.062168] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146842] node 0 deferred pages initialised in 77ms [ 0.148115] devtmpfs: initialized [ 0.149000] x86/mm: Memory block size: 128MB [ 0.151848] gcov: version magic: 0x41383552 [ 0.152643] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157296] pinctrl core: initialized pinctrl subsystem [ 0.158180] [ 0.158812] ************************************************************* [ 0.159013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160017] ** ** [ 0.161015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162016] ** ** [ 0.163018] ** This means that this kernel is built to expose internal ** [ 0.164014] ** IOMMU data structures, which may compromise security on ** [ 0.165016] ** your system. ** [ 0.166016] ** ** [ 0.167027] ** If you see this message and you are not debugging the ** [ 0.168014] ** kernel, report this immediately to your vendor! ** [ 0.169013] ** ** [ 0.170009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171014] ************************************************************* [ 0.172650] NET: Registered protocol family 16 [ 0.173456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177038] cpuidle: using governor menu [ 0.178696] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181448] PCI: Using configuration type 1 for base access [ 0.183092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193141] cryptd: max_cpu_qlen set to 1000 [ 0.194208] ACPI: Added _OSI(Module Device) [ 0.195027] ACPI: Added _OSI(Processor Device) [ 0.196009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196952] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198912] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200512] ACPI: Interpreter enabled [ 0.201055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202010] ACPI: Using IOAPIC for interrupt routing [ 0.203099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213948] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217940] acpiphp: Slot [2] registered [ 0.218117] acpiphp: Slot [5] registered [ 0.219137] acpiphp: Slot [6] registered [ 0.220115] acpiphp: Slot [7] registered [ 0.221113] acpiphp: Slot [8] registered [ 0.222090] acpiphp: Slot [9] registered [ 0.223105] acpiphp: Slot [10] registered [ 0.224096] acpiphp: Slot [3] registered [ 0.225161] acpiphp: Slot [4] registered [ 0.226096] acpiphp: Slot [11] registered [ 0.227116] acpiphp: Slot [12] registered [ 0.228059] acpiphp: Slot [13] registered [ 0.229087] acpiphp: Slot [14] registered [ 0.230096] acpiphp: Slot [15] registered [ 0.231098] acpiphp: Slot [16] registered [ 0.232125] acpiphp: Slot [17] registered [ 0.233089] acpiphp: Slot [18] registered [ 0.234137] acpiphp: Slot [19] registered [ 0.235087] acpiphp: Slot [20] registered [ 0.236116] acpiphp: Slot [21] registered [ 0.237110] acpiphp: Slot [22] registered [ 0.238124] acpiphp: Slot [23] registered [ 0.239026] acpiphp: Slot [24] registered [ 0.240134] acpiphp: Slot [25] registered [ 0.241112] acpiphp: Slot [26] registered [ 0.242083] acpiphp: Slot [27] registered [ 0.243081] acpiphp: Slot [28] registered [ 0.244156] acpiphp: Slot [29] registered [ 0.245071] acpiphp: Slot [30] registered [ 0.246093] acpiphp: Slot [31] registered [ 0.247041] PCI host bridge to bus 0000:00 [ 0.247968] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.248018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.249024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.250017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.251013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.252021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.253132] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.254771] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.255881] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.259886] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261544] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.262012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.263016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.264014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.265388] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266506] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.268031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.270691] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.275011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.284012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.289061] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.292105] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.300020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.311071] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.323013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.335201] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.341012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.348015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.365019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.378437] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.386014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.392015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.406016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.414025] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.421015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.427014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.449021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.459119] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.469018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.477017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.491019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.502100] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.511017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.524017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.543021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.566398] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.569341] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.571365] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.573397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.576204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.580239] iommu: Default domain type: Passthrough [ 0.582500] SCSI subsystem initialized [ 0.584159] ACPI: bus type USB registered [ 0.586100] usbcore: registered new interface driver usbfs [ 0.588089] usbcore: registered new interface driver hub [ 0.590103] usbcore: registered new device driver usb [ 0.592169] pps_core: LinuxPPS API ver. 1 registered [ 0.594012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.597062] PTP clock support registered [ 0.599145] EDAC MC: Ver: 3.0.0 [ 0.601099] PCI: Using ACPI for IRQ routing [ 0.602884] NetLabel: Initializing [ 0.604012] NetLabel: domain hash size = 128 [ 0.606011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.607070] NetLabel: unlabeled traffic allowed by default [ 0.609119] vgaarb: loaded [ 0.610252] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.612015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.621004] clocksource: Switched to clocksource kvm-clock [ 0.725030] VFS: Disk quotas dquot_6.6.0 [ 0.726626] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.729154] *** VALIDATE ramfs *** [ 0.730357] *** VALIDATE hugetlbfs *** [ 0.731786] pnp: PnP ACPI init [ 0.734880] pnp: PnP ACPI: found 6 devices [ 0.756613] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.760203] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.762556] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.764807] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.767278] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.769739] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.772419] NET: Registered protocol family 2 [ 0.774650] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.779436] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.783214] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.788814] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.792376] TCP: Hash tables configured (established 65536 bind 65536) [ 0.795379] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.798775] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.801816] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.805216] NET: Registered protocol family 1 [ 0.807712] RPC: Registered named UNIX socket transport module. [ 0.809995] RPC: Registered udp transport module. [ 0.811902] RPC: Registered tcp transport module. [ 0.813849] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.816806] NET: Registered protocol family 44 [ 0.818037] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.820418] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.822854] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.825325] PCI: CLS 0 bytes, default 64 [ 0.827208] Unpacking initramfs... [ 2.193990] debug: unmapping init [mem 0xffff9febfcc54000-0xffff9febfffbffff] [ 2.198255] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.200724] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.203836] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.713491] Initialise system trusted keyrings [ 2.715184] Key type blacklist registered [ 2.717067] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.727072] zbud: loaded [ 2.730279] *** VALIDATE nfs *** [ 2.731378] *** VALIDATE nfs4 *** [ 2.732680] pstore: using deflate compression [ 2.735686] Platform Keyring initialized [ 2.818891] NET: Registered protocol family 38 [ 2.820261] Key type asymmetric registered [ 2.821538] Asymmetric key parser 'x509' registered [ 2.822794] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.824922] io scheduler mq-deadline registered [ 2.826662] io scheduler kyber registered [ 2.828496] io scheduler bfq registered [ 2.829740] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.831789] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.833856] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.835618] ACPI: Power Button [PWRF] [ 2.839214] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.844424] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.854034] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.858949] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.869658] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.895563] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.921969] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.926297] Non-volatile memory driver v1.3 [ 2.927506] Linux agpgart interface v0.103 [ 2.954050] virtio_blk virtio1: [vda] 145176 512-byte logical blocks (74.3 MB/70.9 MiB) [ 2.957079] vda: detected capacity change from 0 to 74330112 [ 2.973261] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.976266] vdb: detected capacity change from 0 to 1073741824 [ 2.989679] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.992882] vdc: detected capacity change from 0 to 2621440000 [ 3.007772] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.010032] vdd: detected capacity change from 0 to 2621440000 [ 3.024528] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.027277] vde: detected capacity change from 0 to 4294967296 [ 3.044611] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.047851] vdf: detected capacity change from 0 to 4294967296 [ 3.055445] libphy: Fixed MDIO Bus: probed [ 3.061714] usbcore: registered new interface driver usbserial_generic [ 3.063823] usbserial: USB Serial support registered for generic [ 3.066208] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.070662] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.072357] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.074760] mousedev: PS/2 mouse device common for all mice [ 3.077225] rtc_cmos 00:05: RTC can wake from S4 [ 3.080185] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.080568] rtc_cmos 00:05: registered as rtc0 [ 3.084871] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.087454] intel_pstate: CPU model not supported [ 3.089586] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.092853] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.094294] hid: raw HID events driver (C) Jiri Kosina [ 3.097699] usbcore: registered new interface driver usbhid [ 3.099710] usbhid: USB HID core driver [ 3.101279] drop_monitor: Initializing network drop monitor service [ 3.103354] Initializing XFRM netlink socket [ 3.105032] NET: Registered protocol family 10 [ 3.106848] Segment Routing with IPv6 [ 3.108218] NET: Registered protocol family 17 [ 3.111859] mpls_gso: MPLS GSO support [ 3.117227] RAS: Correctable Errors collector initialized. [ 3.120114] AVX version of gcm_enc/dec engaged. [ 3.122313] AES CTR mode by8 optimization enabled [ 3.181705] sched_clock: Marking stable (3181677143, 0)->(4229522907, -1047845764) [ 3.184704] registered taskstats version 1 [ 3.186277] Loading compiled-in X.509 certificates [ 3.187530] zswap: loaded using pool lzo/zbud [ 3.205590] Key type big_key registered [ 3.215553] Key type encrypted registered [ 3.217211] ima: No TPM chip found, activating TPM-bypass! [ 3.219170] ima: Allocated hash algorithm: sha1 [ 3.220746] ima: No architecture policies found [ 3.222289] evm: Initialising EVM extended attributes: [ 3.224020] evm: security.selinux [ 3.225063] evm: security.ima [ 3.225988] evm: security.capability [ 3.227186] evm: HMAC attrs: 0x1 [ 3.229221] rtc_cmos 00:05: setting system clock to 2026-07-07 16:43:19 UTC (1783442599) [ 3.234266] debug: unmapping init [mem 0xffffffff8fc03000-0xffffffff8fdfffff] [ 3.236873] debug: unmapping init [mem 0xffffffff8e982000-0xffffffff8ec58fff] [ 3.245064] Write protecting the kernel read-only data: 28672k [ 3.247821] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 3.250330] debug: unmapping init [mem 0xffffffff8d914000-0xffffffff8d9fffff] [ 3.277595] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.285439] systemd[1]: Detected virtualization kvm. [ 3.286833] systemd[1]: Detected architecture x86-64. [ 3.288477] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.311829] systemd[1]: No hostname configured. [ 3.313550] systemd[1]: Set hostname to . [ 3.315694] random: systemd: uninitialized urandom read (16 bytes read) [ 3.318179] systemd[1]: Initializing machine ID from random generator. [ 3.356119] random: ln: uninitialized urandom read (6 bytes read) [ 3.431788] random: systemd: uninitialized urandom read (16 bytes read) [ 3.434636] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.438551] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.443355] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.945640] device-mapper: uevent: version 1.0.3 [ 3.947903] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.525851] virtio_net virtio0 ens2: renamed from eth0 [ 4.546136] random: fast init done [ 4.580728] scsi host0: ata_piix [ 4.589056] scsi host1: ata_piix [ 4.593278] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.596923] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.163827] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.738878] random: crng init done [ 9.740585] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.988322] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.025920] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.220557] SELinux: Disabled at runtime. [ 11.273978] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.281706] systemd[1]: Detected virtualization kvm. [ 11.284239] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.692359] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.695995] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.701369] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.705192] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.709209] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.716366] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.721574] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ 11.771270] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.098045] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.401484] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.451598] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.565838] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.580395] EDAC sbridge: Ver: 1.1.2 [ 14.017517] Key type dns_resolver registered [ 14.314692] NFS: Registering the id_resolver key type [ 14.316560] Key type id_resolver registered [ 14.318251] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg120-server login: [ 38.840153] libcfs: loading out-of-tree module taints kernel. [ 38.855824] Key type ._llcrypt registered [ 38.856874] Key type .llcrypt registered [ 38.900709] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 54.664952] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 56.188612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.219561] alg: No test for adler32 (adler32-zlib) [ 56.702033] hrtimer: interrupt took 3022781 ns [ 58.292812] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 59.422427] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 61.287249] Key type lgssc registered [ 63.524880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.923158] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.791436] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 148.127316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.170452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.523045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 149.566527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 149.721325] Lustre: lustre-MDT0000: new disk, initializing [ 149.904828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.930742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.885866] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.461284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.560079] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.596842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.608149] Lustre: Skipped 1 previous similar message [ 170.748293] Lustre: lustre-MDT0001: new disk, initializing [ 170.857833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.899794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.934695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.866516] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.337977] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.122193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.488817] Lustre: lustre-OST0000: new disk, initializing [ 193.492335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.496725] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.579788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.758519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.775908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.875918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.330796] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.539240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.675847] Lustre: lustre-OST0001: new disk, initializing [ 216.680089] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.685916] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.746100] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.541321] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.831717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 224.846253] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 224.978551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.710218] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.138989] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.252963] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.685871] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing yml_node [ 263.489270] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 267.167387] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 270.261856] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 272.621663] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jul 7 12:47:46 EDT 2026 [ 293.325229] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 294.947505] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.296059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.315455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.316471] Lustre: Skipped 3 previous similar messages [ 312.333049] Lustre: Skipped 3 previous similar messages [ 317.409844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.420869] Lustre: Skipped 3 previous similar messages [ 317.837894] Lustre: server umount lustre-MDT0000 complete [ 321.746697] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783442918 with bad export cookie 12763154459794136872 [ 321.748042] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.758407] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 322.528082] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.545892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.555060] Lustre: Skipped 1 previous similar message [ 326.560939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.570736] Lustre: Skipped 1 previous similar message [ 331.743817] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.743963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.755613] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 331.791621] Lustre: Skipped 1 previous similar message [ 332.768134] LustreError: 9207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.791301] LustreError: 9207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 336.351296] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.586558] Lustre: server umount lustre-MDT0001 complete [ 340.913420] Lustre: server umount lustre-OST0000 complete [ 344.582359] Lustre: server umount lustre-OST0001 complete [ 350.778027] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 363.731982] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 409.375708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.682862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.723905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.826390] Lustre: lustre-MDT0000: new disk, initializing [ 409.964076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.990523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.601366] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 428.045623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.200559] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 428.276893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 428.282172] Lustre: Skipped 1 previous similar message [ 428.365419] Lustre: lustre-MDT0001: new disk, initializing [ 428.472108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.536132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 428.541361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.260860] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 444.602301] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 450.156853] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 458.296173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 458.711273] Lustre: lustre-OST0000: new disk, initializing [ 458.719912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 458.734829] Lustre: 17057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 458.911229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.211224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 460.245635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 460.462376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.588510] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 476.763754] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 482.008764] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 482.235776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 487.373379] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 487.607872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 490.989619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 490.993360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.016880] Lustre: Skipped 1 previous similar message [ 491.023845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.026980] Lustre: Skipped 2 previous similar messages [ 504.287532] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 504.538339] Lustre: server umount lustre-OST0000 complete [ 510.433257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.440560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.458226] Lustre: Skipped 1 previous similar message [ 510.472194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.486089] Lustre: Skipped 5 previous similar messages [ 515.093810] Lustre: server umount lustre-MDT0000 complete [ 516.577378] LustreError: 15945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.602827] LustreError: 15945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 519.403205] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783443115 with bad export cookie 12763154459794138405 [ 519.419717] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.429398] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 519.813840] Lustre: server umount lustre-MDT0001 complete [ 525.221926] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:51:59 (1783443119) [ 528.353173] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 530.655570] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:52:04 (1783443124) [ 538.541434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.983042] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.103805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.228320] LustreError: 19469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.250947] LustreError: 19469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 544.996037] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 554.468311] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.493263] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 555.695152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.638972] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 570.823985] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 577.113308] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 584.687884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 591.960114] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 600.404369] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 622.053052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 622.068936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.085983] Lustre: Skipped 1 previous similar message [ 622.105159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 622.116035] Lustre: Skipped 2 previous similar messages [ 625.125597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 626.456516] Lustre: server umount lustre-OST0000 complete [ 633.313856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 633.326890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.350089] Lustre: Skipped 1 previous similar message [ 636.968614] Lustre: server umount lustre-MDT0000 complete [ 640.483406] LustreError: 20565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.508882] LustreError: 20565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 640.938536] LustreError: 22271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783443237 with bad export cookie 12763154459794139287 [ 640.944365] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.957596] LustreError: 22271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 641.373958] Lustre: server umount lustre-MDT0001 complete [ 653.539685] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 656.557302] Key type lgssc unregistered [ 656.938812] LNet: 23024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.950321] LNetError: 23024:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.964588] LNet: Removed LNI 192.168.201.120@tcp [ 658.084319] Key type .llcrypt unregistered [ 658.088099] Key type ._llcrypt unregistered [ 688.589631] Key type ._llcrypt registered [ 688.592715] Key type .llcrypt registered [ 688.684953] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 689.904455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.046075] alg: No test for adler32 (adler32-zlib) [ 691.047970] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 691.286769] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 693.015357] Key type lgssc registered [ 694.877504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 705.034056] loop: module loaded [ 705.144878] loop0: detected capacity change from 0 to 204800000 [ 705.478734] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.489852] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.940435] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.667591] loop0: detected capacity change from 0 to 204800000 [ 712.922664] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.936126] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.950574] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 718.935285] loop0: detected capacity change from 0 to 204800000 [ 725.576728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 725.602839] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.035026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 727.119091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 727.247628] Lustre: lustre-MDT0000: new disk, initializing [ 727.318420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.336964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 732.534024] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 741.824105] loop1: detected capacity change from 0 to 204800000 [ 748.093040] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.349211] Lustre: lustre-OST07c6: new disk, initializing [ 748.353810] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 748.359070] Lustre: Skipped 1 previous similar message [ 748.363165] Lustre: 26215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 748.438269] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 754.582196] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 758.350577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 758.368842] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 758.421867] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 760.612392] Lustre: 25202:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 760.627289] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 760.637863] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 760.658460] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 760.674972] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 760.690330] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 778.619563] Lustre: 25201:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 778.627530] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 778.638558] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 778.647701] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 778.654119] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 778.664934] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 783.214407] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 783.222503] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 783.227624] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 783.231596] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.240362] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 783.250401] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.254407] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 783.259346] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.263919] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 783.269586] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.283143] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 783.291590] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.509616] Lustre: 25201:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 786.519518] Lustre: 25201:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 786.534412] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 786.542473] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.550821] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.566689] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.573399] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 786.580328] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.586938] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 786.592925] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.609321] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.620065] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.105133] Lustre: 25201:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 792.118391] Lustre: 25201:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 792.126534] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.133820] Lustre: 25201:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.142334] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.150228] Lustre: 25201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.160116] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 792.171451] Lustre: 25201:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.180854] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 792.191373] Lustre: 25201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.199221] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.206114] Lustre: 25201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.620563] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 803.629814] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 803.635510] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.643507] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 803.649518] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.655598] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 803.661543] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 803.667658] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 803.674876] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 803.683761] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 803.691759] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 803.696322] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 855.525532] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 855.526698] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.553438] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 858.650158] Lustre: server umount lustre-OST07c6 complete [ 868.475984] Lustre: server umount lustre-MDT0000 complete [ 885.645967] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 888.594711] Key type lgssc unregistered [ 888.892448] LNet: 28768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.899646] LNetError: 28768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.915396] LNet: Removed LNI 192.168.201.120@tcp [ 889.676255] Key type .llcrypt unregistered [ 889.677775] Key type ._llcrypt unregistered [ 907.759459] Key type ._llcrypt registered [ 907.763824] Key type .llcrypt registered [ 907.900810] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 908.874988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 908.971832] alg: No test for adler32 (adler32-zlib) [ 910.151380] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 910.507261] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 912.159158] Key type lgssc registered [ 913.172329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 921.004712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 921.050881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.620032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.722633] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 931.811030] LustreError: 30042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.832924] LustreError: 30042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 936.350446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.662884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.843769] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 949.438696] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 955.929795] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 963.161707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.672677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.792066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 969.852451] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 977.815386] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 996.842568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 996.846996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.857378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.871531] Lustre: Skipped 1 previous similar message [ 996.879104] Lustre: Skipped 1 previous similar message [ 1002.098972] LustreError: 31862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.121254] LustreError: 31862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1002.191694] Lustre: server umount lustre-OST0000 complete [ 1007.586257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.604749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.606559] Lustre: Skipped 1 previous similar message [ 1007.626506] Lustre: Skipped 1 previous similar message [ 1012.705806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.711540] Lustre: Skipped 1 previous similar message [ 1017.826962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.851049] Lustre: Skipped 1 previous similar message [ 1020.383262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1020.517212] Lustre: server umount lustre-MDT0000 complete [ 1022.945157] LustreError: 31426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.960950] LustreError: 31426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1024.028149] LustreError: 30020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783443620 with bad export cookie 5751525841881466891 [ 1024.041402] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.051228] LustreError: 30020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1024.228873] Lustre: server umount lustre-MDT0001 complete [ 1034.760729] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1037.634727] Key type lgssc unregistered [ 1037.998176] LNet: 33782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.003902] LNetError: 33782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.022911] LNet: Removed LNI 192.168.201.120@tcp [ 1038.990971] Key type .llcrypt unregistered [ 1038.995425] Key type ._llcrypt unregistered [ 1056.213548] Key type ._llcrypt registered [ 1056.215546] Key type .llcrypt registered [ 1056.316935] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1057.835547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.978409] alg: No test for adler32 (adler32-zlib) [ 1058.980658] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 1059.207614] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1060.871242] Key type lgssc registered [ 1062.009695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1069.686736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1069.704355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.062913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.936303] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1081.314357] LustreError: 35061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.353995] LustreError: 35061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1084.559055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.861426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.125271] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1095.910933] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1101.461936] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1108.458764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.800509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.916715] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1115.951042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1123.055720] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1135.596840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1135.605839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.621237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.610381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.626630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.656097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.731995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.740602] Lustre: Skipped 2 previous similar messages [ 1146.847126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1146.848963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.861556] Lustre: Skipped 1 previous similar message [ 1147.019627] Lustre: server umount lustre-OST0000 complete [ 1151.457706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.465456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.476684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.563776] Lustre: server umount lustre-MDT0000 complete [ 1157.107335] LustreError: 35056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.119842] LustreError: 35056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1160.175348] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783443756 with bad export cookie 6498052718497861529 [ 1160.202471] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.489873] Lustre: server umount lustre-MDT0001 complete [ 1170.517785] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1172.848949] Key type lgssc unregistered [ 1173.145227] LNet: 38596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.153684] LNetError: 38596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.182068] LNet: Removed LNI 192.168.201.120@tcp [ 1175.038912] Key type .llcrypt unregistered [ 1175.043456] Key type ._llcrypt unregistered [ 1195.202841] Key type ._llcrypt registered [ 1195.205707] Key type .llcrypt registered [ 1195.305444] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1196.711201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.731825] alg: No test for adler32 (adler32-zlib) [ 1197.872178] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 1198.077477] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1199.719172] Key type lgssc registered [ 1200.663833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.044126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.058416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.623765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.927849] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1219.043257] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.070707] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1221.708689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.050441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1226.461702] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1233.687035] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1239.276897] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1245.415708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1245.688965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.413322] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1253.897731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1259.548486] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1266.760681] Lustre: server umount lustre-MDT0000 complete [ 1267.698131] LustreError: 41231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.203069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1268.216961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.219536] LustreError: 39992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.220778] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.246140] LustreError: 39992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1269.281931] Lustre: Skipped 1 previous similar message [ 1270.380806] LustreError: 39971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783443866 with bad export cookie 15457361873327769008 [ 1270.385964] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1270.398283] LustreError: 39971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1272.762325] LustreError: 39987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.792407] LustreError: 39987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1272.805249] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.20@tcp (stopping) [ 1274.342422] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.354520] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1275.373519] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1276.793445] Lustre: server umount lustre-MDT0001 complete [ 1297.583537] Lustre: server umount lustre-OST0000 complete [ 1313.427966] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1316.168651] Key type lgssc unregistered [ 1316.454572] LNet: 43852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.458382] LNetError: 43852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.480138] LNet: Removed LNI 192.168.201.120@tcp [ 1318.365554] Key type .llcrypt unregistered [ 1318.367848] Key type ._llcrypt unregistered [ 1338.600407] Key type ._llcrypt registered [ 1338.601945] Key type .llcrypt registered [ 1338.665191] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1339.683673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.699871] alg: No test for adler32 (adler32-zlib) [ 1340.742387] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 1340.912857] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1342.560844] Key type lgssc registered [ 1343.567026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.824460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.849747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.320126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.529794] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1361.888341] LustreError: 45249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.900394] LustreError: 45249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1365.333046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.653847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.876613] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1377.274490] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1382.734868] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1388.955803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.349656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.899979] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1397.548442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1402.413869] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1410.613155] Lustre: lustre-OST0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 1411.052890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1411.171613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.281039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.091483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.095894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.096734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1415.614975] Lustre: lustre-OST0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 1415.642175] Lustre: Skipped 1 previous similar message [ 1416.717244] Lustre: server umount lustre-OST0000 complete [ 1443.808599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.811046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.836710] Lustre: Skipped 1 previous similar message [ 1443.856593] Lustre: Skipped 1 previous similar message [ 1449.574304] Lustre: server umount lustre-MDT0000 complete [ 1454.055525] LustreError: 45250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.086420] LustreError: 45250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1456.168525] LustreError: 45228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783444052 with bad export cookie 11878313648678347153 [ 1456.174659] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.194743] LustreError: 45228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1463.292984] Lustre: server umount lustre-MDT0001 complete [ 1481.028794] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1485.564165] Key type lgssc unregistered [ 1485.898334] LNet: 48960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.904986] LNetError: 48960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.926040] LNet: Removed LNI 192.168.201.120@tcp [ 1487.084451] Key type .llcrypt unregistered [ 1487.086069] Key type ._llcrypt unregistered [ 1521.160771] Key type ._llcrypt registered [ 1521.169817] Key type .llcrypt registered [ 1521.447729] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1523.069680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1523.299685] alg: No test for adler32 (adler32-zlib) [ 1524.620414] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 1524.904421] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1526.560114] Key type lgssc registered [ 1528.069475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.974347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.995633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.801473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.951216] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1548.280920] LustreError: 50378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.308693] LustreError: 50378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1553.378819] LustreError: 50379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.092511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.750907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1562.494974] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1572.571432] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1579.347982] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1588.207760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1588.638415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.178934] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1608.145244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1610.701392] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1618.469291] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1623.259669] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1636.716625] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-0713h10m29s [ 1638.867662] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1642.985220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1642.999915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.061700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.132897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.149372] Lustre: Skipped 4 previous similar messages [ 1645.571370] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 1645.582552] Lustre: Skipped 1 previous similar message [ 1647.792911] Lustre: server umount lustre-MDT0000 complete [ 1650.656944] LustreError: 50378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.209686] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1656.811352] LustreError: 50373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.828731] LustreError: 50373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1660.728646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.852372] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.174139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.284730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1662.198666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1667.732133] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1673.202425] Lustre: DEBUG MARKER: Wait for df (36562) ... [ 1675.229535] Lustre: DEBUG MARKER: done [ 1681.889382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.895564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.907831] LustreError: Skipped 1 previous similar message [ 1681.915566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.955791] Lustre: Skipped 3 previous similar messages [ 1686.963912] Lustre: server umount lustre-OST0000 complete [ 1693.153957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.159562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.189458] Lustre: Skipped 1 previous similar message [ 1693.197418] Lustre: Skipped 3 previous similar messages [ 1698.135751] Lustre: server umount lustre-MDT0000 complete [ 1700.321890] LustreError: 50379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.353180] LustreError: 50379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1702.307965] LustreError: 50358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783444298 with bad export cookie 8704528279764656561 [ 1702.310940] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1702.329890] LustreError: 50358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1702.630117] Lustre: server umount lustre-MDT0001 complete [ 1719.201000] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1723.136846] Key type lgssc unregistered [ 1723.585894] LNet: 55309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.594473] LNetError: 55309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.617313] LNet: Removed LNI 192.168.201.120@tcp [ 1724.625299] Key type .llcrypt unregistered [ 1724.627633] Key type ._llcrypt unregistered [ 1771.387461] Key type ._llcrypt registered [ 1771.389836] Key type .llcrypt registered [ 1771.561696] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1773.534319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.625081] alg: No test for adler32 (adler32-zlib) [ 1774.841425] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 1775.181954] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1776.871152] Key type lgssc registered [ 1778.886940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1788.882589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1788.895880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.345692] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1790.398703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1790.635966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.574812] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1808.179607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.578524] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1808.656064] Lustre: 57522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1809.379356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.734265] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1826.880230] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.865121] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1849.470890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.816334] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1850.097686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.288352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1858.996668] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1868.042222] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1876.952093] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-0713h14m30s [ 1879.938212] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1891.167740] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1896.509040] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1896.607424] LustreError: 57529:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1916.928232] Lustre: lustre-MDT0000: Client 8f0b7fea-6feb-4f37-b512-7053c0f1ff7f (at 192.168.201.20@tcp) reconnecting [ 1936.634340] LustreError: 57529:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1939.841391] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1942.875077] Lustre: DEBUG MARKER: done [ 1950.691387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1950.692209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.703517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.709802] Lustre: Skipped 1 previous similar message [ 1950.724916] Lustre: Skipped 1 previous similar message [ 1955.811308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1955.825801] Lustre: Skipped 1 previous similar message [ 1956.525855] Lustre: server umount lustre-OST0000 complete [ 1962.981633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1962.984987] LustreError: Skipped 1 previous similar message [ 1963.015623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.066163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.178424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.184508] Lustre: Skipped 4 previous similar messages [ 1968.475481] Lustre: server umount lustre-MDT0000 complete [ 1974.592798] LustreError: 57513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783444570 with bad export cookie 625378085892684854 [ 1974.598885] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.604668] LustreError: 57513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1974.857091] Lustre: server umount lustre-MDT0001 complete [ 1987.508736] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1991.007256] Key type lgssc unregistered [ 1991.411125] LNet: 61864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.422978] LNetError: 61864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.456354] LNet: Removed LNI 192.168.201.120@tcp [ 1992.422789] Key type .llcrypt unregistered [ 1992.437214] Key type ._llcrypt unregistered [ 2052.773379] Key type ._llcrypt registered [ 2052.775558] Key type .llcrypt registered [ 2053.022660] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2056.238861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.263269] alg: No test for adler32 (adler32-zlib) [ 2057.509323] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 2057.928495] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2059.727305] Key type lgssc registered [ 2061.454490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.834912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2065.852308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.374417] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2067.416625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2067.706209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.110190] Lustre: Failing over lustre-MDT0000 [ 2069.543375] Lustre: server umount lustre-MDT0000 complete [ 2077.841682] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:17:52 (1783444672) [ 2090.742922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.251870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.579733] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2109.486356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.575641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2109.623544] Lustre: 64669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2109.963774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2115.905427] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2125.541671] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2132.448274] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2144.248558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.734071] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2145.250152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.212785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2155.698454] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2165.310443] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2172.841766] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2181.602252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2181.612968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.633548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.138709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.140142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.160441] Lustre: Skipped 1 previous similar message [ 2188.018702] Lustre: server umount lustre-MDT0000 complete [ 2192.614870] LustreError: 64659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783444788 with bad export cookie 11891996597110483680 [ 2192.621302] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.632029] LustreError: 64659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2193.001357] Lustre: server umount lustre-MDT0001 complete [ 2197.366525] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2203.490732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2210.208113] Lustre: 63588:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783444790/real 1783444790] req@0000000072545c3b x1870077261559424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783444806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.234545] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.262543] Lustre: Skipped 1 previous similar message [ 2214.368193] Lustre: 63590:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783444794/real 1783444794] req@000000009c8b1c31 x1870077261559552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783444810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2215.455210] Lustre: 63590:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783444795/real 1783444795] req@00000000e3821d27 x1870077261559680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783444811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2216.314493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.440038] LustreError: 63587:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000029d86043 x1870077261560448/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2217.844453] LustreError: 66517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.855841] LustreError: 66517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2217.959874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.122457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2218.975404] Lustre: 63588:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783444798/real 1783444798] req@00000000caa87a88 x1870077261559936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783444814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2220.001322] LustreError: 66518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.050885] LustreError: 66518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2223.072079] Lustre: 63589:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783444803/real 1783444803] req@00000000d41c5c57 x1870077261560064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783444819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2224.650599] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2225.122394] LustreError: 66517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.150654] LustreError: 66517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2230.242615] LustreError: 68441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.252373] LustreError: 68441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2235.366037] LustreError: 66518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.379867] LustreError: 66518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2235.687676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.320065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2239.416523] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2242.030441] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2251.069617] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2260.378173] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2272.226556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.246311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.286748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.297985] Lustre: Skipped 1 previous similar message [ 2275.297364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.015126] Lustre: server umount lustre-MDT0000 complete [ 2280.416295] LustreError: 68440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.460672] LustreError: 68440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2282.814429] LustreError: 67583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783444879 with bad export cookie 11891996597110485955 [ 2282.834155] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.843953] LustreError: 67583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2283.567052] Lustre: server umount lustre-MDT0001 complete [ 2292.392966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2300.895080] Lustre: 63589:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783444881/real 1783444881] req@00000000069463fa x1870077261592448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783444897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2300.967138] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.032344] Lustre: Skipped 2 previous similar messages [ 2305.089271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.255944] LustreError: 63587:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000058398ace x1870077261593472/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2309.036207] LustreError: 66518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.208154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.323020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2318.081115] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2330.093913] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2330.111624] Lustre: Skipped 1 previous similar message [ 2333.591155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.328629] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2339.918684] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2350.409836] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2358.201456] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2370.025269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.047340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.072875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.104840] Lustre: Skipped 2 previous similar messages [ 2370.164106] Lustre: Skipped 4 previous similar messages [ 2380.262541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.276880] Lustre: Skipped 5 previous similar messages [ 2381.279160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2381.713176] Lustre: server umount lustre-MDT0000 complete [ 2385.378706] LustreError: 71346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.402058] LustreError: 71346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2388.140619] LustreError: 66525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783444984 with bad export cookie 11891996597110487334 [ 2388.176322] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2388.715938] Lustre: server umount lustre-MDT0001 complete [ 2399.528206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2404.031864] Lustre: DEBUG MARKER: files compared the same [ 2405.857981] Lustre: 63589:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783444986/real 1783444986] req@0000000087d3386e x1870077261624832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783445002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2405.886746] Lustre: 63589:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2405.903522] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.760046] Lustre: server umount lustre-OST0000 complete [ 2431.406925] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2440.786966] Key type lgssc unregistered [ 2441.438421] LNet: 74409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.447704] LNetError: 74409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.470498] LNet: Removed LNI 192.168.201.120@tcp [ 2444.148889] Key type .llcrypt unregistered [ 2444.157780] Key type ._llcrypt unregistered [ 2475.699250] Key type ._llcrypt registered [ 2475.704655] Key type .llcrypt registered [ 2475.963405] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2478.321762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.348097] alg: No test for adler32 (adler32-zlib) [ 2479.642931] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 2480.130238] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2481.903303] Key type lgssc registered [ 2484.175328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2496.398550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2496.423655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.131330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.347543] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug +malloc all [ 2508.270752] LustreError: 75826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.292894] LustreError: 75826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2513.379498] LustreError: 75827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.502564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.067939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2521.246448] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug +malloc all [ 2531.001488] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2538.280254] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2548.417023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2548.862401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.057366] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug +malloc all [ 2567.194974] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2568.305540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2577.887955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.894769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.915527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.913254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2578.918398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.922780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.953870] Lustre: Skipped 1 previous similar message [ 2584.154139] LustreError: 77651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.175367] LustreError: 77651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2584.342494] Lustre: server umount lustre-OST0000 complete [ 2589.670421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.673054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2589.689497] Lustre: Skipped 1 previous similar message [ 2589.703530] Lustre: Skipped 1 previous similar message [ 2594.786414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.807204] Lustre: Skipped 1 previous similar message [ 2596.128355] Lustre: server umount lustre-MDT0000 complete [ 2599.906200] LustreError: 75827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.927824] LustreError: 75827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2601.155669] LustreError: 75806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783445197 with bad export cookie 14194708190651253760 [ 2601.168134] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.173836] LustreError: 75806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2601.946456] Lustre: server umount lustre-MDT0001 complete [ 2617.730337] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2622.002273] Key type lgssc unregistered [ 2622.404279] LNet: 79363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.417801] LNetError: 79363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.471442] LNet: Removed LNI 192.168.201.120@tcp [ 2624.173210] Key type .llcrypt unregistered [ 2624.175532] Key type ._llcrypt unregistered [ 2653.683557] Key type ._llcrypt registered [ 2653.686241] Key type .llcrypt registered [ 2653.763948] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2655.530243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2655.582023] alg: No test for adler32 (adler32-zlib) [ 2656.811197] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 2657.034705] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2658.793884] Key type lgssc registered [ 2660.230341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2669.273978] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2669.297253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2685.087542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2690.207313] LustreError: 80753:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 2716.895202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.180134] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2731.633812] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2739.304828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.388347] Lustre: *** cfs_fail_loc=706, val=0*** [ 2739.715182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.778087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2744.801308] LustreError: 80779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.824642] LustreError: 80779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2746.541072] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2749.926919] LustreError: 80778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.950991] LustreError: 80779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.964101] LustreError: 80779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2756.078848] LustreError: 80778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.115242] LustreError: 80778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2759.018603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.330754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2764.474810] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2774.277741] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2783.201809] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2790.368713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2790.372809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.401584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2790.413567] Lustre: Skipped 1 previous similar message [ 2790.436446] Lustre: Skipped 1 previous similar message [ 2792.388470] Lustre: server umount lustre-OST0000 complete [ 2800.609272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.622732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2800.626114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.641734] Lustre: Skipped 1 previous similar message [ 2803.616896] Lustre: server umount lustre-MDT0000 complete [ 2805.733733] LustreError: 81844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.773579] LustreError: 81844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2808.115548] LustreError: 80760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783445404 with bad export cookie 17748865645497072078 [ 2808.123714] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2808.126906] LustreError: 80760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2808.586582] Lustre: server umount lustre-MDT0001 complete [ 2822.459677] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2826.256376] Key type lgssc unregistered [ 2826.589039] LNet: 84374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2826.600602] LNetError: 84374:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2826.621727] LNet: Removed LNI 192.168.201.120@tcp [ 2827.940659] Key type .llcrypt unregistered [ 2827.943265] Key type ._llcrypt unregistered [ 2868.759491] Key type ._llcrypt registered [ 2868.764949] Key type .llcrypt registered [ 2868.946875] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2871.559494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2871.846517] alg: No test for adler32 (adler32-zlib) [ 2872.954095] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 2873.582518] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2875.263215] Key type lgssc registered [ 2877.793848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2888.080584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2888.100697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.907737] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2903.895401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.333283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2909.668670] LustreError: 86481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.824479] LustreError: 86481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2910.258214] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2913.825303] LustreError: 86479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.844707] LustreError: 86479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2918.882163] LustreError: 86481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.904802] LustreError: 86481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2921.926366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.352491] LustreError: 86485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.403934] LustreError: 86485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2922.427726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.585587] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.609424] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2934.789826] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2939.876165] LustreError: 86481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.900365] LustreError: 86481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2946.784626] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.810812] Lustre: MGS: Client dd0c9eb4-dd90-4af9-8ce4-6ebbec016e4c (at 0@lo) reconnecting [ 2946.840257] LustreError: 86495:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2947.285720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.388823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2953.286778] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2973.162292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.164693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.181433] Lustre: Skipped 1 previous similar message [ 2973.187389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.283146] Lustre: server umount lustre-OST0000 complete [ 2983.394475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2983.394929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.397957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.397967] Lustre: Skipped 1 previous similar message [ 2983.466235] Lustre: Skipped 1 previous similar message [ 2988.513135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2988.527557] Lustre: Skipped 2 previous similar messages [ 2993.634899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2993.638272] Lustre: Skipped 1 previous similar message [ 2996.207096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2996.309507] Lustre: server umount lustre-MDT0000 complete [ 2998.754524] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.809342] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3003.890175] Lustre: server umount lustre-MDT0001 complete [ 3008.467255] Lustre: server umount MGS complete [ 3024.175967] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:33:38 (1783445618) [ 3069.654976] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 3079.918533] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3122.221240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.632753] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3139.705963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.947293] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3140.001242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3140.087205] Lustre: lustre-MDT0001: new disk, initializing [ 3140.221668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3140.254025] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3140.264278] Lustre: 93682:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3140.286480] LustreError: 93682:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3145.423662] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3161.328932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.748056] Lustre: lustre-OST0000: new disk, initializing [ 3161.755524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3161.773287] Lustre: 94448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3161.959288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.417779] Lustre: 94718:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000074ba2e66 x1870078116148096/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783445782 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3171.333863] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3181.979787] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3190.765618] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3190.982823] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.996544] Lustre: MGS: Client 80aa4659-a0e7-425d-9e34-44bf07c34e74 (at 0@lo) reconnecting [ 3191.023555] LustreError: 93668:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3191.106216] Lustre: 95494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3191.166968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3191.316258] Lustre: lustre-MDT0000: new disk, initializing [ 3191.492341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.563941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3192.614494] Lustre: 95518:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004a266d4d x1870078116155520/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783445804 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3197.402855] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3197.800904] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3201.806182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3201.813285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3201.872760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3226.595955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3226.613347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.641541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.395787] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3239.398656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.422209] Lustre: Skipped 6 previous similar messages [ 3239.837801] Lustre: server umount lustre-OST0000 complete [ 3249.632739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.639445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3249.661951] Lustre: Skipped 2 previous similar messages [ 3249.668320] LustreError: Skipped 1 previous similar message [ 3251.687794] Lustre: server umount lustre-MDT0000 complete [ 3254.754456] LustreError: 93654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.780140] LustreError: 93654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3263.233233] Lustre: server umount lustre-MDT0001 complete [ 3268.655258] Lustre: server umount MGS complete [ 3286.367180] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:38:00 (1783445880) [ 3304.275770] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3306.825591] Key type lgssc unregistered [ 3307.085389] LNet: 98020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3307.091204] LNetError: 98020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3307.111868] LNet: Removed LNI 192.168.201.120@tcp [ 3307.923757] Key type .llcrypt unregistered [ 3307.928118] Key type ._llcrypt unregistered [ 3326.807644] Key type ._llcrypt registered [ 3326.811033] Key type .llcrypt registered [ 3326.971249] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3328.211763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3328.401915] alg: No test for adler32 (adler32-zlib) [ 3329.528810] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 3329.709469] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3331.410830] Key type lgssc registered [ 3332.915576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3341.250587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3341.292599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.815739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.808996] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3353.059651] LustreError: 99244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.089457] LustreError: 99244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3357.389184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.822847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3363.047570] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3370.952615] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3377.391815] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3384.504826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.751720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.838726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3390.640790] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3398.051692] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3408.353855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3408.360202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.368256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3412.449400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3412.464406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3412.477059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.575326] Lustre: server umount lustre-OST0000 complete [ 3419.105726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3419.115712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.131878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.134796] Lustre: Skipped 1 previous similar message [ 3422.689464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.691201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.678777] Lustre: server umount lustre-MDT0000 complete [ 3427.810120] LustreError: 99245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.841131] LustreError: 99245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3427.949777] LustreError: 99224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446024 with bad export cookie 1566520278087931797 [ 3427.954832] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.975146] LustreError: 99224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3428.274957] Lustre: server umount lustre-MDT0001 complete [ 3440.814632] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3443.909863] Key type lgssc unregistered [ 3444.229710] LNet: 102776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3444.239214] LNetError: 102776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3444.252172] LNet: Removed LNI 192.168.201.120@tcp [ 3445.132478] Key type .llcrypt unregistered [ 3445.135440] Key type ._llcrypt unregistered [ 3459.557996] Key type ._llcrypt registered [ 3459.561253] Key type .llcrypt registered [ 3459.646357] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3460.774453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3460.815578] alg: No test for adler32 (adler32-zlib) [ 3461.841886] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 3462.057210] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3463.767949] Key type lgssc registered [ 3465.049677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3470.913608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3470.919641] LustreError: 103850:0:(tgt_mount.c:2568:server_fill_super()) cfs_race id 716 sleeping [ 3470.932143] LustreError: 103853:0:(tgt_mount.c:2568:server_fill_super()) cfs_fail_race id 716 waking [ 3470.936579] LustreError: 103850:0:(tgt_mount.c:2568:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3470.943103] LustreError: 103850:0:(tgt_mount.c:2580:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3470.948056] LustreError: 103850:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3470.953871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.626364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.918700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.261216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3484.713581] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3490.153116] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing lsmod [ 3492.977390] LustreError: 105056:0:(tgt_mount.c:2568:server_fill_super()) cfs_race id 716 sleeping [ 3493.025368] LustreError: 105060:0:(tgt_mount.c:2568:server_fill_super()) cfs_fail_race id 716 waking [ 3493.033715] LustreError: 105056:0:(tgt_mount.c:2568:server_fill_super()) cfs_fail_race id 716 awake: rc=4951 [ 3493.042328] LustreError: 105056:0:(tgt_mount.c:2580:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3493.055225] LustreError: 105056:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3493.060287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3493.497888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.979181] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.988991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.995356] Lustre: Skipped 2 previous similar messages [ 3499.004985] Lustre: Skipped 2 previous similar messages [ 3502.826303] Lustre: server umount lustre-MDT0000 complete [ 3504.102541] LustreError: 103884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.117670] LustreError: 103884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3505.130279] LustreError: 103891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.151359] LustreError: 103891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3506.737205] LustreError: 103869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446103 with bad export cookie 8458607150797131905 [ 3506.742773] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3506.745135] LustreError: 103869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3507.223337] LustreError: 104322:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002b3eaad2 x1870078733733632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3507.487902] Lustre: server umount lustre-MDT0001 complete [ 3521.076817] Lustre: server umount lustre-OST0000 complete [ 3529.421039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.825605] LustreError: 106122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3529.876455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.635182] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3535.340319] LustreError: 106122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.406933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.448173] LustreError: 106123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.710395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.362464] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3551.305300] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3556.307993] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3562.878977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3563.193453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.538248] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3575.527975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3576.405534] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3586.022408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3586.026702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.033577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.151886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.160518] Lustre: Skipped 1 previous similar message [ 3596.255121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3596.260043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.264266] Lustre: Skipped 1 previous similar message [ 3596.450481] Lustre: server umount lustre-OST0000 complete [ 3601.379347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.384162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3601.401297] Lustre: Skipped 2 previous similar messages [ 3601.434235] Lustre: Skipped 2 previous similar messages [ 3606.549536] LustreError: 107218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3606.571774] LustreError: 107218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3606.627410] Lustre: server umount lustre-MDT0000 complete [ 3610.176709] LustreError: 106104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446206 with bad export cookie 8458607150797132605 [ 3610.182740] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3610.190055] LustreError: 106104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3610.557662] Lustre: server umount lustre-MDT0001 complete [ 3621.611207] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3624.349405] Key type lgssc unregistered [ 3624.656060] LNet: 109656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.660901] LNetError: 109656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3624.672491] LNet: Removed LNI 192.168.201.120@tcp [ 3625.441883] Key type .llcrypt unregistered [ 3625.444213] Key type ._llcrypt unregistered [ 3647.936932] Key type ._llcrypt registered [ 3647.939806] Key type .llcrypt registered [ 3648.096304] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3649.625446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3649.691112] alg: No test for adler32 (adler32-zlib) [ 3650.734355] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 3650.950059] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3652.615244] Key type lgssc registered [ 3653.873890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3662.043634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3662.064435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.657207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.035722] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3673.062253] LustreError: 111077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.089237] LustreError: 111077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3676.720625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.049028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3681.526633] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3688.339754] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3693.566206] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3700.072691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3700.551182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.500591] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3707.770554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3714.181775] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3720.378520] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3723.966864] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3728.355525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.364720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3728.365125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3728.371680] Lustre: Skipped 1 previous similar message [ 3730.044338] Lustre: server umount lustre-OST0000 complete [ 3733.473128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.489373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3733.489817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3733.495879] Lustre: Skipped 1 previous similar message [ 3733.522315] Lustre: Skipped 2 previous similar messages [ 3738.594344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3738.597469] Lustre: Skipped 1 previous similar message [ 3739.700736] Lustre: server umount lustre-MDT0000 complete [ 3743.249785] LustreError: 113608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446339 with bad export cookie 5905909399863114912 [ 3743.252887] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3743.259850] LustreError: 113608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3743.491757] Lustre: server umount lustre-MDT0001 complete [ 3752.944628] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3755.536202] Key type lgssc unregistered [ 3755.806956] LNet: 114715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.816608] LNetError: 114715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3755.833104] LNet: Removed LNI 192.168.201.120@tcp [ 3756.835515] Key type .llcrypt unregistered [ 3756.839675] Key type ._llcrypt unregistered [ 3771.960409] Key type ._llcrypt registered [ 3771.963362] Key type .llcrypt registered [ 3772.070489] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3773.074989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3773.097767] alg: No test for adler32 (adler32-zlib) [ 3774.296516] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 3774.529358] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3776.160846] Key type lgssc registered [ 3777.165271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3784.263750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3784.278120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.679976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.565644] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3794.915709] LustreError: 115915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.930472] LustreError: 115915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3797.086666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.411424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3801.014071] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3807.112208] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3811.109684] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3816.524419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3816.772147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3822.271622] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3828.591714] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3830.061542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3840.480524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3840.485319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.491179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3840.493564] Lustre: Skipped 1 previous similar message [ 3845.492375] Lustre: server umount lustre-OST0000 complete [ 3848.675537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3848.679960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.689570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3848.693441] Lustre: Skipped 1 previous similar message [ 3850.721088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.727172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.735433] Lustre: Skipped 1 previous similar message [ 3854.487459] Lustre: server umount lustre-MDT0000 complete [ 3855.848315] LustreError: 118363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.882670] LustreError: 118363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3857.851303] LustreError: 115895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446454 with bad export cookie 13426780193590539056 [ 3857.869065] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3857.878291] LustreError: 115895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3858.224274] Lustre: server umount lustre-MDT0001 complete [ 3870.088603] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3872.679370] Key type lgssc unregistered [ 3873.025974] LNet: 119447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3873.035180] LNetError: 119447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3873.054436] LNet: Removed LNI 192.168.201.120@tcp [ 3874.080972] Key type .llcrypt unregistered [ 3874.082932] Key type ._llcrypt unregistered [ 3892.206974] Key type ._llcrypt registered [ 3892.208421] Key type .llcrypt registered [ 3892.260839] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3892.921101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3892.951349] alg: No test for adler32 (adler32-zlib) [ 3893.933965] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 3894.118818] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3895.759843] Key type lgssc registered [ 3896.379627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3901.812916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3901.826340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.088499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.092229] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3912.028516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.299406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3915.520439] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3920.868720] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3924.540819] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3929.389356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3929.658683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.031418] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3938.991117] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3948.076244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3953.601248] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3956.049795] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3956.055762] Lustre: Skipped 2 previous similar messages [ 3959.262120] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3959.269139] Lustre: Skipped 2 previous similar messages [ 3962.938230] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3962.955951] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3966.202940] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.20@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3966.208435] Lustre: Skipped 2 previous similar messages [ 3971.620269] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.20@tcp 192.168.0.[2,10]@tcp [ 3976.137372] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.20/24@tcp in log lustre-MDT0000 [ 3976.140342] Lustre: Skipped 2 previous similar messages [ 3981.908123] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.20/24@tcp [ 3981.913102] Lustre: Skipped 2 previous similar messages [ 3987.937688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.942234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.958343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.962141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.969048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.976250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.983622] Lustre: Skipped 1 previous similar message [ 3992.726630] Lustre: server umount lustre-OST0000 complete [ 3999.200871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.216519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.219171] Lustre: Skipped 1 previous similar message [ 4001.583593] Lustre: server umount lustre-MDT0000 complete [ 4004.172563] LustreError: 120831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446600 with bad export cookie 3795889418042338377 [ 4004.175852] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.181161] LustreError: 120831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4004.421521] Lustre: server umount lustre-MDT0001 complete [ 4011.858936] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4013.715398] Key type lgssc unregistered [ 4014.017726] LNet: 125850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4014.025858] LNetError: 125850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4014.040469] LNet: Removed LNI 192.168.201.120@tcp [ 4014.762567] Key type .llcrypt unregistered [ 4014.771278] Key type ._llcrypt unregistered [ 4035.007915] Key type ._llcrypt registered [ 4035.014111] Key type .llcrypt registered [ 4035.084712] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4035.647780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.673359] alg: No test for adler32 (adler32-zlib) [ 4036.633653] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 4036.783030] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4038.415170] Key type lgssc registered [ 4039.130693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4044.203541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4044.218722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.532725] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4045.536876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4045.547535] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.20/24@tcp [ 4048.459466] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4054.165491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.387172] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4054.419227] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.20/24@tcp [ 4054.434799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.549779] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4062.806790] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4067.635654] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4072.804159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4073.139489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.666233] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4084.053384] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4091.580234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4102.117217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.118763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4102.119298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4102.134715] Lustre: Skipped 1 previous similar message [ 4106.940634] Lustre: server umount lustre-OST0000 complete [ 4110.816724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4110.827614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.840800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.845061] Lustre: Skipped 1 previous similar message [ 4112.362339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4112.366198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.725414] Lustre: server umount lustre-MDT0000 complete [ 4117.477681] LustreError: 127390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.494035] LustreError: 127390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4118.325546] LustreError: 128129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446714 with bad export cookie 14322768427439468058 [ 4118.332909] LustreError: 128129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4118.333095] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4118.640517] Lustre: server umount lustre-MDT0001 complete [ 4126.029683] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4127.855740] Key type lgssc unregistered [ 4128.024589] LNet: 131003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4128.029431] LNetError: 131003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4128.045306] LNet: Removed LNI 192.168.201.120@tcp [ 4128.563703] Key type .llcrypt unregistered [ 4128.565679] Key type ._llcrypt unregistered [ 4146.042917] Key type ._llcrypt registered [ 4146.045025] Key type .llcrypt registered [ 4146.098635] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4155.948190] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4156.587983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4156.657058] alg: No test for adler32 (adler32-zlib) [ 4157.602637] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 4157.797985] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4159.423173] Key type lgssc registered [ 4160.137654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4184.425709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4184.449673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4185.636239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4185.668626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4185.729502] Lustre: lustre-MDT0000: new disk, initializing [ 4185.785458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.805216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4188.546934] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4196.367116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.502442] Lustre: 134377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4196.558873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4196.562702] Lustre: Skipped 1 previous similar message [ 4196.617910] Lustre: lustre-MDT0001: new disk, initializing [ 4196.702549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4196.741763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4196.750120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4199.652787] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4206.857572] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4211.070307] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4215.546748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4215.755489] Lustre: lustre-OST0000: new disk, initializing [ 4215.758603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4215.763166] Lustre: 136346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4215.824667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.307344] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4225.634054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4225.644862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4225.741519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4227.395032] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4230.549427] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4230.723683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4233.493096] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4233.645682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4236.260298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.263401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4236.277103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4236.284017] Lustre: Skipped 1 previous similar message [ 4241.327419] Lustre: server umount lustre-OST0000 complete [ 4246.500491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.518468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4246.521746] Lustre: Skipped 2 previous similar messages [ 4250.068635] Lustre: server umount lustre-MDT0000 complete [ 4252.873273] LustreError: 135174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446849 with bad export cookie 7427690728883292436 [ 4252.880123] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4252.884430] LustreError: 135174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4253.205597] Lustre: server umount lustre-MDT0001 complete [ 4259.180996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.494814] LustreError: 138564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.503336] LustreError: 138564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4259.549140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.693367] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4264.932064] LustreError: 138565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.025733] LustreError: 138564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.223942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.493581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.830453] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4277.991151] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4281.893734] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4286.602906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.813289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.977859] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4295.664463] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4298.400851] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4304.731405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 4307.016503] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4307.158801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4314.661668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.805174] Lustre: lustre-OST0001: new disk, initializing [ 4314.808820] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4314.813736] Lustre: 141978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4314.885769] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4318.469189] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4324.065436] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4324.402716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4324.427724] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4324.476400] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4325.309794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4328.234814] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4328.418761] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4332.093995] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d86c5ce9000.ost_server_uuid 50 [ 4333.293390] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d86c5ce9000.ost_server_uuid in FULL state after 0 sec [ 4339.689825] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.695667] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4339.701058] Lustre: Skipped 1 previous similar message [ 4339.708367] Lustre: Skipped 2 previous similar messages [ 4344.617828] Lustre: server umount lustre-OST0001 complete [ 4344.801930] LustreError: 140385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.818194] LustreError: 140385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4349.927522] LustreError: 140386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.927816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.939332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4349.945199] LustreError: 140386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4349.979050] Lustre: Skipped 1 previous similar message [ 4353.310821] Lustre: server umount lustre-OST0000 complete [ 4356.576246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4356.580917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.595067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4356.597909] Lustre: Skipped 1 previous similar message [ 4361.891802] Lustre: server umount lustre-MDT0000 complete [ 4364.177640] LustreError: 138546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783446960 with bad export cookie 7427690728883293318 [ 4364.181662] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4364.185721] LustreError: 138546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4364.425980] Lustre: server umount lustre-MDT0001 complete [ 4375.837412] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4377.446385] Key type lgssc unregistered [ 4377.609777] LNet: 144982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4377.613608] LNetError: 144982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4377.625144] LNet: Removed LNI 192.168.201.120@tcp [ 4378.135385] Key type .llcrypt unregistered [ 4378.137091] Key type ._llcrypt unregistered [ 4403.335429] Key type ._llcrypt registered [ 4403.336894] Key type .llcrypt registered [ 4403.397961] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4413.123209] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4413.743204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4413.825701] alg: No test for adler32 (adler32-zlib) [ 4414.810214] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 4414.986410] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4416.615268] Key type lgssc registered [ 4417.463800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4438.990303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4439.002054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.147270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4440.173471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4440.225395] Lustre: lustre-MDT0000: new disk, initializing [ 4440.282253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.294284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4443.304437] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4451.004199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4451.055244] Lustre: 149111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4451.091977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4451.094537] Lustre: Skipped 1 previous similar message [ 4451.140819] Lustre: lustre-MDT0001: new disk, initializing [ 4451.187127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4451.211305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4451.218757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4454.153810] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4461.258933] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4464.798980] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4469.234019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4469.449090] Lustre: lustre-OST0000: new disk, initializing [ 4469.451151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4469.455517] Lustre: 151077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4469.516366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.252701] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4477.992867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4478.000526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4478.079322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4479.774369] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4483.766552] Lustre: Failing over lustre-OST0000 [ 4483.908758] Lustre: server umount lustre-OST0000 complete [ 4499.651242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4499.854212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4499.885659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.901390] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4502.927582] Lustre: 147335:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783447083/real 1783447083] req@00000000b6a2f1df x1870079733037312/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1783447099 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4502.994523] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4503.552191] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4503.968111] Lustre: 147337:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783447084/real 1783447084] req@0000000074ba2e66 x1870079733037568/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783447100 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4503.981367] Lustre: 147337:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4505.086995] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4505.089271] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4505.927776] Lustre: Failing over lustre-MDT0000 [ 4506.110229] Lustre: server umount lustre-MDT0000 complete [ 4507.616497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4507.627366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.642289] Lustre: Skipped 1 previous similar message [ 4509.217471] Lustre: 147338:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783447089/real 1783447089] req@000000003425ea9b x1870079733039616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783447105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4509.232618] Lustre: 147338:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4510.180428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.186828] Lustre: Skipped 1 previous similar message [ 4513.212253] LustreError: 149116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.224760] LustreError: 149116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4514.335817] Lustre: 147338:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783447094/real 1783447094] req@0000000009aa6295 x1870079733040768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783447110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4515.302853] LustreError: 149121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.312868] LustreError: 149121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4518.328336] LustreError: 151815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.418423] LustreError: 149116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.429744] LustreError: 149116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4521.688312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4521.780554] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4521.791470] Lustre: 149113:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4522.005767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.040196] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4523.457490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4524.819178] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4527.088799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4527.096156] Lustre: Skipped 1 previous similar message [ 4527.104610] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4527.131335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4530.656859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4530.660968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.666951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.200711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.207748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4532.208636] Lustre: Skipped 1 previous similar message [ 4535.486377] Lustre: server umount lustre-OST0000 complete [ 4542.434965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.435113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4542.437619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4542.446956] Lustre: Skipped 2 previous similar messages [ 4547.553786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.559803] Lustre: Skipped 2 previous similar messages [ 4552.159155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4552.288808] Lustre: server umount lustre-MDT0000 complete [ 4552.672552] LustreError: 150297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.683255] LustreError: 150297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4554.522346] LustreError: 149102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783447150 with bad export cookie 16368125790697839227 [ 4554.525904] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4554.527801] LustreError: 149102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4554.722031] Lustre: server umount lustre-MDT0001 complete [ 4561.129386] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 4562.960431] Key type lgssc unregistered [ 4563.148877] LNet: 154727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4563.156242] LNetError: 154727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4563.175875] LNet: Removed LNI 192.168.201.120@tcp [ 4563.742591] Key type .llcrypt unregistered [ 4563.745047] Key type ._llcrypt unregistered [ 4592.288299] Key type ._llcrypt registered [ 4592.290090] Key type .llcrypt registered [ 4592.362982] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4593.166617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4593.173732] alg: No test for adler32 (adler32-zlib) [ 4594.129080] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 4594.308075] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 4595.919446] Key type lgssc registered [ 4596.773918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4602.233386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4602.244103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.595969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4606.493868] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4613.094462] LustreError: 156246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.107094] LustreError: 156246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4613.142049] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4616.159057] LustreError: 156274:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4616.166699] LustreError: 156274:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4616.172659] LustreError: 156274:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 4616.360202] Lustre: server umount lustre-MDT0000 complete [ 4625.546466] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4631.808253] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4651.781934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.898946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4651.914208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4651.957358] Lustre: lustre-MDT0000: new disk, initializing [ 4651.996768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4652.005606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4654.457533] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4660.496046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.560180] Lustre: 159575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4660.600319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4660.602988] Lustre: Skipped 1 previous similar message [ 4660.641945] Lustre: lustre-MDT0001: new disk, initializing [ 4660.689954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4660.713147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4660.717716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4663.018645] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4668.543935] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4671.918645] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4675.883690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.063962] Lustre: lustre-OST0000: new disk, initializing [ 4676.066757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4676.072484] Lustre: 161551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4676.133142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.908942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4677.912234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4677.981561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4679.524316] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4685.512978] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4689.627805] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4690.957478] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:01:26 (1783447286) [ 4847.071559] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.071995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4847.078263] Lustre: Skipped 2 previous similar messages [ 4847.082348] Lustre: Skipped 1 previous similar message [ 4852.192515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4852.197903] Lustre: Skipped 3 previous similar messages [ 4852.803484] Lustre: server umount lustre-MDT0000 complete [ 4854.789821] LustreError: 160377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783447451 with bad export cookie 7166266536776727027 [ 4854.790803] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4854.795220] LustreError: 160377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4857.311622] LustreError: 159581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.311670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.332205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4859.359373] LustreError: 159582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.359396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4864.479695] LustreError: 159581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.481464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4869.087119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4869.177129] Lustre: server umount lustre-MDT0001 complete [ 4877.406838] Lustre: server umount lustre-OST0000 complete [ 4881.812449] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4886.181349] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4902.436772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.574977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4902.613369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4902.653452] Lustre: lustre-MDT0000: new disk, initializing [ 4902.705797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4902.713318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4904.384930] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4909.746611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.797623] Lustre: 166193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4909.829340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4909.832926] Lustre: Skipped 1 previous similar message [ 4909.875545] Lustre: lustre-MDT0001: new disk, initializing [ 4909.929143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4909.933746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4911.531459] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4916.352880] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4918.898699] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4921.648075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4921.780554] Lustre: lustre-OST0000: new disk, initializing [ 4921.782980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4921.786533] Lustre: 168162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4921.819499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4921.822115] Lustre: Skipped 1 previous similar message [ 4924.352909] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4928.877340] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4931.025184] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4932.099391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4932.103386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4932.148330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4933.172887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4935.104713] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4935.205312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4937.187608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4937.189661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.191311] LustreError: Skipped 1 previous similar message [ 4937.202063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4937.206975] Lustre: Skipped 2 previous similar messages [ 4938.166163] Lustre: server umount lustre-OST0000 complete [ 4940.768305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4940.771771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.778183] Lustre: Skipped 1 previous similar message [ 4946.095286] Lustre: server umount lustre-MDT0000 complete [ 4947.429771] LustreError: 166201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.443992] LustreError: 166201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4947.883765] LustreError: 166186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783447544 with bad export cookie 7166266536776885857 [ 4947.885646] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.890982] LustreError: 166186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4948.054366] Lustre: server umount lustre-MDT0001 complete [ 4952.096354] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:05:47 (1783447547) [ 4959.082334] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4963.605970] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4977.840716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4977.938905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4977.954049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4977.993516] Lustre: lustre-MDT0000: new disk, initializing [ 4978.027849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.035397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4979.555316] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4984.684980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.748439] Lustre: 172925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4984.886044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4986.599133] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4991.103085] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4993.579023] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4996.583215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4996.726430] Lustre: 174901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4999.060054] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4999.559436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4999.603826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5003.716330] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5009.888780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5009.891174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5009.892862] LustreError: Skipped 1 previous similar message [ 5009.897400] Lustre: Skipped 1 previous similar message [ 5009.900639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5009.906402] Lustre: Skipped 4 previous similar messages [ 5012.567982] Lustre: server umount lustre-OST0000 complete [ 5020.469126] Lustre: server umount lustre-MDT0000 complete [ 5022.337291] LustreError: 172915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783447618 with bad export cookie 7166266536776886718 [ 5022.341081] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.342641] LustreError: 172915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5022.473748] Lustre: server umount lustre-MDT0001 complete [ 5026.527112] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:07:02 (1783447622) [ 5034.622750] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5039.636694] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5054.012458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.110211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5054.125493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5054.127904] Lustre: Skipped 3 previous similar messages [ 5054.166481] Lustre: lustre-MDT0000: new disk, initializing [ 5054.167848] Lustre: Skipped 2 previous similar messages [ 5054.198990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.201541] Lustre: Skipped 2 previous similar messages [ 5054.206973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5054.210036] Lustre: Skipped 2 previous similar messages [ 5055.828976] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5060.738318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.791884] Lustre: 179216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5060.917265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5062.748289] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5067.666520] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5070.005947] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5073.029239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.196816] Lustre: 181190:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5075.821811] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5080.670341] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5082.708168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5092.835259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5092.836724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.842527] LustreError: Skipped 1 previous similar message [ 5092.844106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5092.850799] Lustre: Skipped 3 previous similar messages [ 5092.859289] Lustre: Skipped 5 previous similar messages [ 5094.842350] Lustre: server umount lustre-OST0000 complete [ 5096.929212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5111.263120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5111.342090] Lustre: server umount lustre-MDT0000 complete [ 5112.924190] LustreError: 181189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783447709 with bad export cookie 7166266536776887810 [ 5112.927323] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5112.928609] LustreError: 181189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5113.077448] Lustre: server umount lustre-MDT0001 complete [ 5118.111985] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 5119.606303] Key type lgssc unregistered [ 5119.769156] LNet: 183046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5119.773212] LNetError: 183046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5119.789886] LNet: Removed LNI 192.168.201.120@tcp [ 5120.242552] Key type .llcrypt unregistered [ 5120.243891] Key type ._llcrypt unregistered [ 5131.737631] Key type ._llcrypt registered [ 5131.739261] Key type .llcrypt registered [ 5131.801639] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5132.173474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5132.207236] alg: No test for adler32 (adler32-zlib) [ 5133.121540] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 5133.256573] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 5134.856176] Key type lgssc registered [ 5135.434802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5138.807482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5138.816882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.017070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.789607] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5146.151549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5146.314310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5148.646462] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5158.385552] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5165.078797] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5172.795120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5173.492816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.153546] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5184.776421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5193.943702] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5215.720590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5215.736413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.756255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5216.737393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5216.740515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5216.745352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.762849] Lustre: Skipped 1 previous similar message [ 5218.074705] Lustre: server umount lustre-OST0000 complete [ 5223.396602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5223.411573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.436916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5226.987936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5226.989136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.096386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5232.109918] Lustre: Skipped 2 previous similar messages [ 5237.215610] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5237.403346] Lustre: server umount lustre-MDT0000 complete [ 5241.609649] LustreError: 184419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783447837 with bad export cookie 8505422312165572809 [ 5241.611457] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5241.622266] LustreError: 184419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5242.142831] Lustre: server umount lustre-MDT0001 complete [ 5252.351774] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 5254.865417] Key type lgssc unregistered [ 5255.170037] LNet: 187999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5255.173389] LNetError: 187999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5255.194225] LNet: Removed LNI 192.168.201.120@tcp [ 5256.141625] Key type .llcrypt unregistered [ 5256.143238] Key type ._llcrypt unregistered [ 5275.581573] Key type ._llcrypt registered [ 5275.582943] Key type .llcrypt registered [ 5275.685345] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5276.702869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5276.759692] alg: No test for adler32 (adler32-zlib) [ 5277.729853] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 5277.916288] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 5279.583162] Key type lgssc registered [ 5280.415707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5286.513512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5286.530913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.993560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5291.939735] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5298.145819] LustreError: 189397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.155705] LustreError: 189397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5299.644375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.035215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5304.124980] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5311.277500] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5316.156600] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5322.012684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5322.280098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5327.354617] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5333.567917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5333.766157] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5340.640786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5340.642175] Lustre: lustre-OST0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 5340.652028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.725646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.728868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.728922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.728928] Lustre: Skipped 1 previous similar message [ 5344.914780] Lustre: server umount lustre-OST0000 complete [ 5350.877583] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d86c732f800.ost_server_uuid 50 [ 5352.146055] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5366.753314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5366.765414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.779953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5366.783644] Lustre: Skipped 1 previous similar message [ 5368.864673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5368.870107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.562393] Lustre: server umount lustre-MDT0000 complete [ 5373.454120] LustreError: 189378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783447969 with bad export cookie 17505183473931190316 [ 5373.464427] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5373.712692] Lustre: server umount lustre-MDT0001 complete [ 5380.542514] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:12:55 (1783447975) [ 5385.770358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.137645] LustreError: 193308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.210681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5389.415835] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5391.331740] LustreError: 193309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.780420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.026093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5399.445893] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5405.067966] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5409.411436] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.205649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.455446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.865884] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5426.760695] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5431.845696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5432.182074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5432.407266] Lustre: lustre-OST0001: new disk, initializing [ 5432.410528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5432.415523] Lustre: 196147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5432.496356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5437.516120] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5441.611528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5441.622905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5441.743807] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5445.716616] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5451.733581] Lustre: lustre-OST0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 5451.738308] Lustre: Skipped 1 previous similar message [ 5452.277049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5452.285561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.465907] Lustre: server umount lustre-OST0000 complete [ 5456.832170] LustreError: 195129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.516283] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5459.693529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5461.945494] LustreError: 195129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5461.962627] LustreError: 195129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5463.496954] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5463.704218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5472.186586] LustreError: 195128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5472.211454] LustreError: 195128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5477.865788] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.874781] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5477.887897] Lustre: Skipped 2 previous similar messages [ 5477.890476] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5477.895377] Lustre: Skipped 3 previous similar messages [ 5477.905450] LustreError: Skipped 1 previous similar message [ 5483.070437] Lustre: server umount lustre-OST0001 complete [ 5493.258178] LustreError: 194108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.258338] Lustre: server umount lustre-MDT0000 complete [ 5493.268768] LustreError: 194108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5496.890062] LustreError: 193290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783448093 with bad export cookie 17505183473931191387 [ 5496.893579] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5496.902191] LustreError: 193290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5497.210250] Lustre: server umount lustre-MDT0001 complete [ 5520.119097] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:15:15 (1783448115) [ 5526.255418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.423103] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5526.443927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5526.572456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.646846] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5538.333247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.432785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5538.478936] Lustre: 199585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5542.860120] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5549.560180] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5555.282463] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5561.663054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5561.883118] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5567.989334] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5571.744151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5575.363416] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5581.498879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.540916] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5587.136165] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5591.833316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5594.021846] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5597.157129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.169530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5597.170817] Lustre: Skipped 2 previous similar messages [ 5597.171668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5597.182076] Lustre: Skipped 5 previous similar messages [ 5602.457666] Lustre: server umount lustre-OST0000 complete [ 5604.796331] LustreError: 201437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.812603] LustreError: 201437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5617.645134] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5622.837387] Lustre: server umount lustre-OST0001 complete [ 5632.992112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5632.999373] Lustre: Skipped 9 previous similar messages [ 5640.161140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5640.357104] Lustre: server umount lustre-MDT0000 complete [ 5643.228690] LustreError: 200345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783448239 with bad export cookie 17505183473931192794 [ 5643.231043] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5643.236225] LustreError: 200345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5643.496831] Lustre: server umount lustre-MDT0001 complete [ 5661.936492] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:17:37 (1783448257) [ 5675.712638] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5683.939108] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5715.230491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.438282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5715.469115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5715.521350] Lustre: lustre-MDT0000: new disk, initializing [ 5715.628464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5715.632894] Lustre: Skipped 3 previous similar messages [ 5715.644675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5719.340969] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5728.476665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.576606] Lustre: 207678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5728.619537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5728.624091] Lustre: Skipped 1 previous similar message [ 5728.706476] Lustre: lustre-MDT0001: new disk, initializing [ 5728.834706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5728.847851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5732.259861] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5739.811701] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5744.516944] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5750.029278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5750.265723] Lustre: lustre-OST0000: new disk, initializing [ 5750.270149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5750.277212] Lustre: 209642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5755.500945] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5755.946837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5755.953619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5756.059820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5763.047571] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5767.219243] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5767.426451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5771.310503] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5771.568606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5776.355832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.358191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5776.368656] Lustre: Skipped 6 previous similar messages [ 5776.380973] Lustre: Skipped 3 previous similar messages [ 5779.169523] Lustre: server umount lustre-OST0000 complete [ 5785.060367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5785.065630] LustreError: Skipped 2 previous similar messages [ 5796.319125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5798.466064] Lustre: server umount lustre-MDT0000 complete [ 5800.928771] LustreError: 207688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.949525] LustreError: 207688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5801.207356] LustreError: 208471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783448397 with bad export cookie 17505183473931194572 [ 5801.211186] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.213967] LustreError: 208471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5801.474947] Lustre: server umount lustre-MDT0001 complete [ 5808.013511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.435695] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5816.922088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.864178] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5824.108775] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5827.453864] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5831.774384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5836.713796] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5842.173248] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5845.539424] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5845.727389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5849.032993] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5849.179935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5852.644320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5852.644591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.656593] Lustre: Skipped 2 previous similar messages [ 5856.734658] Lustre: server umount lustre-OST0000 complete [ 5861.461631] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5864.840305] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5892.034431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5892.271369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5892.274931] Lustre: Skipped 5 previous similar messages [ 5893.286956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.298149] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.306106] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5893.317616] Lustre: Skipped 1 previous similar message [ 5896.581496] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5902.013580] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5911.698611] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 5916.426947] Lustre: server umount lustre-OST0000 complete [ 5919.209995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.213884] Lustre: Skipped 10 previous similar messages [ 5929.497181] Lustre: server umount lustre-MDT0000 complete [ 5931.974821] LustreError: 211843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783448528 with bad export cookie 17505183473931195461 [ 5931.979610] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5931.985121] LustreError: 211843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5942.382324] Lustre: server umount lustre-MDT0001 complete [ 5948.274558] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:22:23 (1783448543) [ 5953.532942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.862525] LustreError: 217827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.878417] LustreError: 217827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5957.015887] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5963.278599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.777134] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5972.319933] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5976.762840] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5981.819925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5986.846850] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5992.126509] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5995.241267] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6000.713816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 6003.645133] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6003.774705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6008.233564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.387795] Lustre: lustre-OST0001: new disk, initializing [ 6008.389993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6008.395215] Lustre: 221231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6011.587718] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6015.025296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6015.032324] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6015.099859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6017.142510] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6019.942802] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6020.081732] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6023.014825] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6023.178908] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6025.184345] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6025.192975] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6025.201630] Lustre: Skipped 4 previous similar messages [ 6030.715953] Lustre: server umount lustre-OST0001 complete [ 6035.651398] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6035.876926] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6039.069836] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6039.204597] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6064.948203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6068.555620] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6070.245235] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6070.258992] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6070.260937] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6070.272300] Lustre: Skipped 1 previous similar message [ 6073.188785] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6074.252974] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6081.609573] Lustre: server umount lustre-OST0001 complete [ 6101.529450] Lustre: server umount lustre-OST0000 complete [ 6118.367312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6118.458069] Lustre: server umount lustre-MDT0000 complete [ 6120.697554] LustreError: 221230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783448716 with bad export cookie 17505183473931196413 [ 6120.698736] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.702337] LustreError: 221230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6126.991223] Lustre: server umount lustre-MDT0001 complete [ 6140.607940] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:25:36 (1783448736) [ 6144.823566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.988397] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6145.004212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6147.652199] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6153.061113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.121664] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6153.139854] Lustre: 226469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6153.272750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6153.276435] Lustre: Skipped 6 previous similar messages [ 6155.887501] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6159.931094] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6163.150625] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6166.859321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6166.991447] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6170.606101] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6175.163712] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6180.572508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6180.649364] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6183.659648] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6187.166880] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6188.003668] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6190.045247] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6190.159955] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6192.292196] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6192.388275] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6194.946020] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d86c6ece000.ost_server_uuid 50 [ 6195.732446] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d86c6ece000.ost_server_uuid in FULL state after 0 sec [ 6196.260973] Lustre: Permanently deactivating lustre-OST0001 [ 6196.270473] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6197.466846] Lustre: Permanently reactivating lustre-OST0001 [ 6201.824970] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6201.827962] Lustre: Skipped 21 previous similar messages [ 6205.005638] Lustre: server umount lustre-OST0001 complete [ 6212.584218] LustreError: 228308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6212.596835] LustreError: 228308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 6221.279061] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6221.353904] Lustre: server umount lustre-OST0000 complete [ 6229.226842] Lustre: server umount lustre-MDT0000 complete [ 6231.223549] LustreError: 229337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783448827 with bad export cookie 17505183473931197701 [ 6231.227522] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6231.229726] LustreError: 229337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6231.425245] Lustre: server umount lustre-MDT0001 complete [ 6244.583120] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:27:19 (1783448839) [ 6248.611167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.765191] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6248.793188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6248.797925] Lustre: Skipped 5 previous similar messages [ 6251.205817] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6255.461907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.528742] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6255.540602] Lustre: 233151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6257.842662] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6261.575166] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6264.348924] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6267.789131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6267.918247] Lustre: Permanently deactivating lustre-OST0000 [ 6267.925225] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6270.915966] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6274.507821] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6275.624048] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6275.635033] LustreError: 235746:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6275.646564] LustreError: 235746:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6278.600243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6281.990891] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6285.540617] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6285.895148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6291.530062] Lustre: Permanently reactivating lustre-OST0000 [ 6291.538695] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6291.542552] Lustre: Skipped 2 previous similar messages [ 6293.191654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6293.198312] Lustre: Skipped 12 previous similar messages [ 6293.201031] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6293.206225] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.211425] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6293.229360] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6293.238769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6296.035775] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6296.039829] LustreError: Skipped 7 previous similar messages [ 6301.445707] Lustre: server umount lustre-OST0001 complete [ 6309.605225] Lustre: server umount lustre-OST0000 complete [ 6317.806536] Lustre: server umount lustre-MDT0000 complete [ 6319.641610] LustreError: 233894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783448915 with bad export cookie 17505183473931199332 [ 6319.644279] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6319.651141] LustreError: 233894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6319.884197] Lustre: server umount lustre-MDT0001 complete [ 6325.537763] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6326.922105] Key type lgssc unregistered [ 6327.076497] LNet: 238003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6327.080180] LNetError: 238003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6327.088315] LNet: Removed LNI 192.168.201.120@tcp [ 6327.494614] Key type .llcrypt unregistered [ 6327.496491] Key type ._llcrypt unregistered [ 6339.752533] Key type ._llcrypt registered [ 6339.754079] Key type .llcrypt registered [ 6339.805821] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6340.217418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6340.232366] alg: No test for adler32 (adler32-zlib) [ 6341.126645] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 6341.272508] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6342.895400] Key type lgssc registered [ 6343.457827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6348.661395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6348.676880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.921142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.042479] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6356.344901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.415889] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6356.440987] Lustre: 239456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.447057] Lustre: Permanently deactivating lustre-MDT0001 [ 6356.456569] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6356.683799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6357.881164] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6357.885515] LustreError: 239499:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6357.895075] LustreError: 239499:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6357.902931] LustreError: 239499:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 6358.522940] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6361.550647] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6364.205350] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6367.873853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6368.052861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6370.587993] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6374.297542] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6375.208468] LustreError: 241306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.218619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6375.219898] LustreError: 241306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6377.956304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.081492] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6380.853514] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6381.171686] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6381.180750] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6384.817259] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6386.875757] Lustre: Permanently reactivating lustre-MDT0001 [ 6386.893552] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6386.898166] Lustre: Skipped 2 previous similar messages [ 6393.806059] Lustre: lustre-MDT0001: Client c4e6009f-3045-4c63-a693-56a7fe4c8330 (at 192.168.201.20@tcp) reconnecting [ 6397.148066] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6398.062107] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6400.580500] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6400.679638] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6402.797461] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6402.927263] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6404.370318] Lustre: Permanently deactivating lustre-MDT0001 [ 6404.377048] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6404.383029] Lustre: Skipped 2 previous similar messages [ 6410.254792] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6418.401178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.414412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6418.415926] Lustre: Skipped 1 previous similar message [ 6421.303536] Lustre: server umount lustre-MDT0000 complete [ 6423.213119] LustreError: 239451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449019 with bad export cookie 5919656277183464645 [ 6423.217418] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6423.222040] LustreError: 239451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6423.377558] Lustre: server umount lustre-MDT0001 complete [ 6431.109424] Lustre: server umount lustre-OST0000 complete [ 6436.464480] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:30:32 (1783449032) [ 6442.106706] Lustre: server umount lustre-OST0001 complete [ 6445.442114] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 6450.541212] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6467.895625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.060729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6468.064910] Lustre: Skipped 2 previous similar messages [ 6468.079856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6468.119610] Lustre: lustre-MDT0000: new disk, initializing [ 6468.161464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.172092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6469.975350] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6475.350458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.416716] Lustre: 247717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6475.448326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6475.450229] Lustre: Skipped 1 previous similar message [ 6475.498563] Lustre: lustre-MDT0001: new disk, initializing [ 6475.559972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6475.565682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6477.351355] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6481.679273] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6484.007546] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6487.096898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6487.215737] Lustre: lustre-OST0000: new disk, initializing [ 6487.217827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6487.221225] Lustre: 249683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6487.262594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.266813] Lustre: Skipped 1 previous similar message [ 6489.716966] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6493.701435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6493.704746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6493.770548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6494.711961] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6495.855199] LustreError: 247724:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6500.927255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.007233] Lustre: lustre-OST0001: new disk, initializing [ 6501.010116] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6501.013038] Lustre: 250797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6502.931284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6502.935031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6502.978769] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6503.651880] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6505.855636] LustreError: 247724:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6505.877462] LustreError: 247725:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6508.283806] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6509.036491] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6515.975168] LustreError: 247725:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6518.226463] Lustre: lustre-OST0001: Not available for connect from 192.168.201.20@tcp (stopping) [ 6518.233081] Lustre: Skipped 3 previous similar messages [ 6518.240853] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.247936] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.249362] LustreError: Skipped 1 previous similar message [ 6518.262105] Lustre: Skipped 3 previous similar messages [ 6522.951160] Lustre: server umount lustre-OST0001 complete [ 6523.321336] LustreError: 249667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6528.484513] LustreError: 249668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6528.493228] LustreError: 249668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6533.607206] LustreError: 251568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.614496] LustreError: 251568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6538.721733] LustreError: 249668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6538.724712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.732805] LustreError: 249668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6538.754694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.756305] Lustre: Skipped 2 previous similar messages [ 6543.841665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6543.846794] Lustre: Skipped 2 previous similar messages [ 6548.963939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6548.964170] LustreError: 249667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.966876] Lustre: Skipped 1 previous similar message [ 6548.974706] LustreError: 249667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6549.983165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6550.065939] Lustre: server umount lustre-OST0000 complete [ 6552.545117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6552.547600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.552466] Lustre: Skipped 1 previous similar message [ 6559.200731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6559.203697] Lustre: Skipped 4 previous similar messages [ 6566.367109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6566.478554] Lustre: server umount lustre-MDT0000 complete [ 6568.142392] LustreError: 247711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449164 with bad export cookie 5919656277183467235 [ 6568.145868] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.147289] LustreError: 247711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6574.355638] Lustre: server umount lustre-MDT0001 complete [ 6579.448686] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6580.725546] Key type lgssc unregistered [ 6580.923860] LNet: 252832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6580.929316] LNetError: 252832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6580.947602] LNet: Removed LNI 192.168.201.120@tcp [ 6581.383239] Key type .llcrypt unregistered [ 6581.384733] Key type ._llcrypt unregistered [ 6599.677550] Key type ._llcrypt registered [ 6599.679526] Key type .llcrypt registered [ 6599.733692] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6600.236691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6600.272615] alg: No test for adler32 (adler32-zlib) [ 6601.172375] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 6601.285502] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6602.895181] Key type lgssc registered [ 6603.372490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6606.800171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6606.807041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.914974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6607.924546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6607.989821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.879643] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6613.589113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.643281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6613.661737] Lustre: 255018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6613.769869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.409553] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6618.830222] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6621.213720] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6624.648773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.803153] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6624.943942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.550520] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6631.968774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6632.053115] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6677.988157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6677.989639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.003807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6678.007156] Lustre: Skipped 1 previous similar message [ 6680.040330] Lustre: server umount lustre-OST0000 complete [ 6682.022994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6688.083769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6688.251505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.228558] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6694.925361] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6698.468208] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6698.477483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6698.477899] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6698.490204] Lustre: Skipped 1 previous similar message [ 6734.307856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.308511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6734.309992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6734.316205] Lustre: Skipped 2 previous similar messages [ 6734.330484] LustreError: Skipped 2 previous similar messages [ 6735.469729] Lustre: server umount lustre-OST0000 complete [ 6739.424473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.429578] Lustre: Skipped 1 previous similar message [ 6739.432522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6739.434565] Lustre: Skipped 2 previous similar messages [ 6743.661777] Lustre: server umount lustre-MDT0000 complete [ 6745.610638] LustreError: 255758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449341 with bad export cookie 13706670607353491442 [ 6745.611047] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6745.616065] LustreError: 255758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6745.791390] Lustre: server umount lustre-MDT0001 complete [ 6750.847578] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6752.168382] Key type lgssc unregistered [ 6752.318540] LNet: 260439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6752.322170] LNetError: 260439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6752.337046] LNet: Removed LNI 192.168.201.120@tcp [ 6752.738117] Key type .llcrypt unregistered [ 6752.739990] Key type ._llcrypt unregistered [ 6764.575567] Key type ._llcrypt registered [ 6764.577456] Key type .llcrypt registered [ 6764.631817] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6764.999294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6765.072244] alg: No test for adler32 (adler32-zlib) [ 6765.954442] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 6766.054807] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6767.655321] Key type lgssc registered [ 6768.170214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6771.668898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6771.680334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.895499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.784874] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6778.926798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.132732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6781.140930] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6784.651227] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6787.690949] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6790.900844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6791.120419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6794.234856] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6797.655643] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6798.311442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6819.809589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6819.812227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.815042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6819.818060] Lustre: Skipped 1 previous similar message [ 6819.822875] Lustre: Skipped 1 previous similar message [ 6824.947898] Lustre: server umount lustre-OST0000 complete [ 6830.053658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.056424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6830.068043] Lustre: Skipped 1 previous similar message [ 6835.170888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6835.175252] Lustre: Skipped 2 previous similar messages [ 6840.288173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6841.311119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6841.384400] Lustre: server umount lustre-MDT0000 complete [ 6843.209214] LustreError: 261813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449439 with bad export cookie 17214543562952316077 [ 6843.212387] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6843.219726] LustreError: 261813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6843.380780] Lustre: server umount lustre-MDT0001 complete [ 6848.280137] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6849.529247] Key type lgssc unregistered [ 6849.712689] LNet: 266156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6849.718519] LNetError: 266156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6849.731233] LNet: Removed LNI 192.168.201.120@tcp [ 6850.151441] Key type .llcrypt unregistered [ 6850.153461] Key type ._llcrypt unregistered [ 6858.842565] Key type ._llcrypt registered [ 6858.844876] Key type .llcrypt registered [ 6858.900539] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6859.457058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6859.471744] alg: No test for adler32 (adler32-zlib) [ 6860.375636] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 6860.467610] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6862.055157] Key type lgssc registered [ 6862.633843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6866.326542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6866.342530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.576616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.565779] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6873.220922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.424115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6875.291752] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6878.389146] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6880.781516] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6883.568927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6883.714505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6886.453762] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6889.817269] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6892.014428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6900.194578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.196654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6900.198561] Lustre: Skipped 1 previous similar message [ 6900.204247] Lustre: Skipped 1 previous similar message [ 6905.314510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6905.316805] Lustre: Skipped 1 previous similar message [ 6905.695910] Lustre: server umount lustre-OST0000 complete [ 6909.411472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6909.415318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.420764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6910.432297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6913.372990] Lustre: server umount lustre-MDT0000 complete [ 6914.851451] LustreError: 267336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449511 with bad export cookie 17666545685844832346 [ 6914.856167] LustreError: 267336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6914.856881] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6915.052138] Lustre: server umount lustre-MDT0001 complete [ 6919.995761] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 6921.452306] Key type lgssc unregistered [ 6921.621806] LNet: 271030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6921.628412] LNetError: 271030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6921.640403] LNet: Removed LNI 192.168.201.120@tcp [ 6922.069086] Key type .llcrypt unregistered [ 6922.071122] Key type ._llcrypt unregistered [ 6932.734356] Key type ._llcrypt registered [ 6932.736019] Key type .llcrypt registered [ 6932.776434] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 6933.224492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6933.245136] alg: No test for adler32 (adler32-zlib) [ 6934.114114] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 6934.224138] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 6935.815218] Key type lgssc registered [ 6936.236135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6939.607754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6939.615777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.812817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.330233] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6945.817383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.987483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6947.574449] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6950.300261] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6952.323749] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6954.926040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.083370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.325696] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 6960.024260] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6966.373956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6982.114869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.116106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6982.122216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6982.124830] LustreError: Skipped 1 previous similar message [ 6982.128063] Lustre: Skipped 1 previous similar message [ 6986.342076] Lustre: server umount lustre-OST0000 complete [ 6992.353124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.354078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6992.354094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6992.360507] Lustre: Skipped 2 previous similar messages [ 6994.075623] Lustre: server umount lustre-MDT0000 complete [ 6995.498544] LustreError: 272406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449591 with bad export cookie 5312516893237700487 [ 6995.502273] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6995.506625] LustreError: 272406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6995.633347] Lustre: server umount lustre-MDT0001 complete [ 6999.932757] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7001.150453] Key type lgssc unregistered [ 7001.319791] LNet: 276748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7001.324716] LNetError: 276748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7001.334637] LNet: Removed LNI 192.168.201.120@tcp [ 7001.688728] Key type .llcrypt unregistered [ 7001.690455] Key type ._llcrypt unregistered [ 7010.015754] Key type ._llcrypt registered [ 7010.016839] Key type .llcrypt registered [ 7010.069040] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7010.543634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7010.555067] alg: No test for adler32 (adler32-zlib) [ 7011.497780] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 7011.633742] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7013.247161] Key type lgssc registered [ 7013.735436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7017.619552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7017.629025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7018.893648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7020.914111] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7024.687871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.874813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7026.519231] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7029.653095] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7031.983481] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7034.498774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.629947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7036.751181] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7039.581890] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7043.878909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7049.184944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7049.186130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.186846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7049.196176] Lustre: Skipped 1 previous similar message [ 7054.308091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7054.312289] Lustre: Skipped 2 previous similar messages [ 7055.317284] Lustre: server umount lustre-OST0000 complete [ 7059.425401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.426929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7059.439859] Lustre: Skipped 1 previous similar message [ 7059.445237] Lustre: Skipped 1 previous similar message [ 7063.186302] Lustre: server umount lustre-MDT0000 complete [ 7064.544997] LustreError: 277967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.554960] LustreError: 277967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7065.104254] LustreError: 277927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449661 with bad export cookie 13184800031872535040 [ 7065.108655] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7065.111451] LustreError: 277927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7065.285205] Lustre: server umount lustre-MDT0001 complete [ 7069.621274] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7070.668324] Key type lgssc unregistered [ 7070.798440] LNet: 281656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7070.801250] LNetError: 281656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7070.810375] LNet: Removed LNI 192.168.201.120@tcp [ 7071.145553] Key type .llcrypt unregistered [ 7071.146831] Key type ._llcrypt unregistered [ 7075.397724] Key type ._llcrypt registered [ 7075.399125] Key type .llcrypt registered [ 7075.445177] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7082.069901] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7089.359041] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7089.769078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7089.810201] alg: No test for adler32 (adler32-zlib) [ 7090.685144] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 7090.787802] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7092.375414] Key type lgssc registered [ 7092.808348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7106.610873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7106.621577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.720154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7107.730981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7107.766659] Lustre: lustre-MDT0000: new disk, initializing [ 7107.786211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7107.793100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7109.237814] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7113.652873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.694080] Lustre: 285179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7113.719019] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7113.721101] Lustre: Skipped 1 previous similar message [ 7113.755549] Lustre: lustre-MDT0001: new disk, initializing [ 7113.795867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7113.809892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7113.817598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7115.219224] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7119.280862] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7121.309339] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7123.936869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.036064] Lustre: lustre-OST0000: new disk, initializing [ 7124.037583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7124.040981] Lustre: 287151:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7124.071984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.795595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7124.799718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7124.834219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7126.137318] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7130.203423] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7131.852539] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7131.941553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7133.613442] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7133.691081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7135.202443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7135.202529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.207695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7135.211624] Lustre: Skipped 1 previous similar message [ 7135.215434] Lustre: Skipped 1 previous similar message [ 7140.321821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7140.324406] Lustre: Skipped 1 previous similar message [ 7140.495949] Lustre: server umount lustre-OST0000 complete [ 7144.416041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7144.418962] LustreError: Skipped 1 previous similar message [ 7144.420577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.426930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7145.442065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.133799] Lustre: server umount lustre-MDT0000 complete [ 7151.533620] LustreError: 285172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449747 with bad export cookie 17693851916685733193 [ 7151.534700] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7151.538743] LustreError: 285172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7151.651209] Lustre: server umount lustre-MDT0001 complete [ 7154.902390] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:42:30 (1783449750) [ 7157.479212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.643946] LustreError: 289560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.650240] LustreError: 289560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7157.669253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.142080] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7162.380307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.547319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7164.120387] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7166.837875] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7168.912120] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7171.277315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7171.410948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7173.743963] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7176.394454] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7193.058629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7193.059112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.062103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7193.064922] Lustre: Skipped 1 previous similar message [ 7193.067061] Lustre: Skipped 2 previous similar messages [ 7196.505973] Lustre: server umount lustre-OST0000 complete [ 7198.209034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.210212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7198.213882] Lustre: Skipped 1 previous similar message [ 7198.219486] Lustre: Skipped 1 previous similar message [ 7204.326051] Lustre: server umount lustre-MDT0000 complete [ 7205.918203] LustreError: 291401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449802 with bad export cookie 17693851916685734061 [ 7205.921450] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7205.923083] LustreError: 291401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7206.044183] Lustre: server umount lustre-MDT0001 complete [ 7210.224584] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7211.402534] Key type lgssc unregistered [ 7211.547361] LNet: 293115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7211.550270] LNetError: 293115:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7211.558337] LNet: Removed LNI 192.168.201.120@tcp [ 7211.925564] Key type .llcrypt unregistered [ 7211.926836] Key type ._llcrypt unregistered [ 7221.774208] Key type ._llcrypt registered [ 7221.775519] Key type .llcrypt registered [ 7221.818986] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7227.927429] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7228.324580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7228.384100] alg: No test for adler32 (adler32-zlib) [ 7229.253526] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 7229.355965] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7230.943205] Key type lgssc registered [ 7231.359996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7248.930607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7248.938339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.028957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7250.039382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7250.079853] Lustre: lustre-MDT0000: new disk, initializing [ 7250.108476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.115047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7251.563763] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7256.321039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.370617] Lustre: 296911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7256.396117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7256.398973] Lustre: Skipped 1 previous similar message [ 7256.442469] Lustre: lustre-MDT0001: new disk, initializing [ 7256.489383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.509282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7256.513681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7257.998345] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7261.851260] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7263.858778] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7266.185402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7266.294688] Lustre: lustre-OST2710: new disk, initializing [ 7266.297776] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7266.302185] Lustre: 298881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7266.338189] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7268.655585] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7272.768283] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7273.468523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7273.473369] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7273.511984] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7275.317737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7275.384787] Lustre: lustre-OST03e8: new disk, initializing [ 7275.387749] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7275.392042] Lustre: 299944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7275.424767] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7277.759075] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7280.894803] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7281.168036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7281.172577] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7281.209951] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7285.175927] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7285.261406] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7286.985677] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7287.085761] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7288.882821] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7288.972388] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7290.618318] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7290.713228] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7292.384095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7292.387212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.392851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7296.480368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7296.480956] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7296.489209] Lustre: Skipped 1 previous similar message [ 7298.089401] Lustre: server umount lustre-MDT0000 complete [ 7299.671642] LustreError: 296904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449895 with bad export cookie 12540361573982887468 [ 7299.680030] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.807059] Lustre: server umount lustre-MDT0001 complete [ 7310.791212] Lustre: server umount lustre-OST2710 complete [ 7322.551097] Lustre: server umount lustre-OST03e8 complete [ 7330.052885] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7333.660489] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7347.353519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.440922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7347.452479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7347.489674] Lustre: lustre-MDT0000: new disk, initializing [ 7347.510744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7347.518941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7348.992101] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7354.006611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7354.049322] Lustre: 305187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7354.153129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7355.773198] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7359.771646] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7361.935243] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7364.815806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7364.964158] Lustre: lustre-OST0000: new disk, initializing [ 7364.965559] Lustre: Skipped 1 previous similar message [ 7364.967097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7364.968828] Lustre: Skipped 2 previous similar messages [ 7364.970668] Lustre: 307151:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7364.999608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.002724] Lustre: Skipped 1 previous similar message [ 7367.515634] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7371.768696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7371.771776] Lustre: Skipped 1 previous similar message [ 7371.774111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7371.809315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7371.942714] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7373.627604] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7373.714260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7375.452226] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7375.551635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7376.865765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7376.866594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.869731] LustreError: Skipped 1 previous similar message [ 7376.874514] Lustre: Skipped 1 previous similar message [ 7376.877819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7376.881138] Lustre: Skipped 4 previous similar messages [ 7378.460685] Lustre: server umount lustre-OST0000 complete [ 7379.937120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7379.940784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7379.947055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7387.104320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7387.108336] Lustre: Skipped 3 previous similar messages [ 7394.273045] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7394.340165] Lustre: server umount lustre-MDT0000 complete [ 7395.806915] LustreError: 305179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783449992 with bad export cookie 12540361573982889134 [ 7395.810378] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7395.812117] LustreError: 305179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7395.933218] Lustre: server umount lustre-MDT0001 complete [ 7399.496197] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:46:35 (1783449995) [ 7400.152265] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7400.813090] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:46:36 (1783449996) [ 7402.218422] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7412.981260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.086354] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7413.093358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7413.151245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7414.598849] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7417.733662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.784803] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7417.796371] Lustre: 311005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7419.381881] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7435.814831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.913849] Lustre: Denying initial registration attempt from nid 192.168.201.120@tcp, specified as failover [ 7435.918961] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7435.922973] LustreError: 312458:0:(tgt_mount.c:2612:server_fill_super()) Unable to start targets: -99 [ 7435.926455] LustreError: 312458:0:(tgt_mount.c:2130:server_put_super()) no obd lustre-OST0000 [ 7435.929193] LustreError: 312458:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7435.955433] Lustre: server umount lustre-OST0000 complete [ 7435.957652] LustreError: 312458:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7442.400251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.405045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7442.407237] Lustre: Skipped 2 previous similar messages [ 7442.409848] Lustre: Skipped 2 previous similar messages [ 7444.637285] Lustre: server umount lustre-MDT0000 complete [ 7446.152225] LustreError: 311747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450042 with bad export cookie 12540361573982890002 [ 7446.155427] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7446.156880] LustreError: 311747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7446.268231] Lustre: server umount lustre-MDT0001 complete [ 7450.697151] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7451.918870] Key type lgssc unregistered [ 7452.084729] LNet: 313577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7452.088115] LNetError: 313577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7452.095487] LNet: Removed LNI 192.168.201.120@tcp [ 7452.505576] Key type .llcrypt unregistered [ 7452.507637] Key type ._llcrypt unregistered [ 7456.796186] Key type ._llcrypt registered [ 7456.797371] Key type .llcrypt registered [ 7456.853424] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7457.256322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7457.313084] alg: No test for adler32 (adler32-zlib) [ 7458.173680] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 7458.277187] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7459.855176] Key type lgssc registered [ 7460.243867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7485.732475] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7489.199846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7489.206658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.286119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7490.299774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7490.364385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7491.925269] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7495.074256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.107793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7495.121693] Lustre: 316317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7495.232127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7496.809303] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7499.492791] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7501.667288] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7517.218237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7517.286972] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7517.358162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.421332] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7521.970519] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7522.794737] Lustre: server umount lustre-OST0000 complete [ 7525.856093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7525.859070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.863926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7526.385850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7526.386143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.270354] Lustre: server umount lustre-MDT0000 complete [ 7531.563199] LustreError: 316307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450127 with bad export cookie 6092699608721004269 [ 7531.565359] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7531.568136] LustreError: 316307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7531.682619] Lustre: server umount lustre-MDT0001 complete [ 7535.700557] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7536.792304] Key type lgssc unregistered [ 7536.923327] LNet: 319878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7536.929313] LNetError: 319878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7536.939574] LNet: Removed LNI 192.168.201.120@tcp [ 7537.247744] Key type .llcrypt unregistered [ 7537.248784] Key type ._llcrypt unregistered [ 7547.117123] Key type ._llcrypt registered [ 7547.118200] Key type .llcrypt registered [ 7547.163855] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7547.616501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7547.626149] alg: No test for adler32 (adler32-zlib) [ 7548.487941] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 7548.583513] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7550.175167] Key type lgssc registered [ 7550.542577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7553.233081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7553.242363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.422702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7555.957345] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7559.431924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7559.577813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7561.374254] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7564.590767] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7566.664120] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7571.686595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7571.884611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7574.579459] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7577.674210] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7589.319428] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [ 7589.349161] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.351758] LustreError: 321271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.358332] Lustre: Skipped 2 previous similar messages [ 7589.372774] LustreError: 321271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7589.387566] Lustre: server umount lustre-MDT0000 complete [ 7591.188535] LustreError: 321252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450187 with bad export cookie 15570016979964999435 [ 7591.192689] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7591.197512] LustreError: 321252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7594.425079] LustreError: 321265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.433093] LustreError: 321265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7594.438776] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.20@tcp (stopping) [ 7594.464671] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.515354] LustreError: 323711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7596.515590] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7596.527528] Lustre: Skipped 1 previous similar message [ 7596.531250] LustreError: 323711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7597.362770] Lustre: server umount lustre-MDT0001 complete [ 7599.889195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7603.528664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.970558] LustreError: 320582:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000093303abf x1870083018218624/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7616.130434] LustreError: 323100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7616.168434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7616.195484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7618.195388] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7621.282031] LustreError: 324856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7621.295068] LustreError: 324856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7622.137077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7622.310084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.345855] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7626.405996] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7627.376255] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7630.063375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7632.145682] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7635.823500] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7637.984464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7637.987509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7637.991753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7642.082612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7642.083922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.094332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7643.349657] Lustre: server umount lustre-OST0000 complete [ 7647.207058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.213861] Lustre: Skipped 1 previous similar message [ 7650.906904] Lustre: server umount lustre-MDT0000 complete [ 7652.250323] LustreError: 321253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450248 with bad export cookie 15570016979965006589 [ 7652.252873] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7652.255137] LustreError: 321253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7652.389329] Lustre: server umount lustre-MDT0001 complete [ 7656.470867] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7657.705600] Key type lgssc unregistered [ 7657.855334] LNet: 327529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7657.860541] LNetError: 327529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7657.870596] LNet: Removed LNI 192.168.201.120@tcp [ 7658.264325] Key type .llcrypt unregistered [ 7658.265941] Key type ._llcrypt unregistered [ 7669.222401] Key type ._llcrypt registered [ 7669.224151] Key type .llcrypt registered [ 7669.275854] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7669.727777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7669.764793] alg: No test for adler32 (adler32-zlib) [ 7670.661930] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 7670.766204] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7672.359161] Key type lgssc registered [ 7672.813078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7676.411699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7676.421070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.643113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7679.533186] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7683.525713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7683.709763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7685.776519] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7688.677826] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7690.880404] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7692.255991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.257765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7692.262331] Lustre: Skipped 1 previous similar message [ 7693.282384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.288789] Lustre: Skipped 2 previous similar messages [ 7698.332490] Lustre: server umount lustre-MDT0000 complete [ 7698.400933] LustreError: 328924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.410135] LustreError: 328924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7699.797650] LustreError: 328904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450296 with bad export cookie 2289217280258878833 [ 7699.800283] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7699.802060] LustreError: 328904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7699.934723] Lustre: server umount lustre-MDT0001 complete [ 7705.997812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.155194] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7706.168490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7706.247818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.865508] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7711.560802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.606958] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7711.622890] Lustre: 331509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7711.656600] LustreError: 331519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.758265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7713.683297] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7716.833723] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7719.110556] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7724.452322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.529154] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7724.533360] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7724.537167] LustreError: 333390:0:(tgt_mount.c:2612:server_fill_super()) Unable to start targets: -2 [ 7724.540100] LustreError: 333390:0:(tgt_mount.c:2130:server_put_super()) no obd lustre-OST0000 [ 7724.542590] LustreError: 333390:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7724.568329] Lustre: server umount lustre-OST0000 complete [ 7724.570036] LustreError: 333390:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7729.960138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7730.094803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7730.186088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7730.586567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7732.575768] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7735.605398] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7738.590072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7738.622250] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7738.658355] Lustre: lustre-OST0001: new disk, initializing [ 7738.660333] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7738.663342] Lustre: 334930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7738.711886] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7740.834252] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7743.583798] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7744.847471] Lustre: server umount lustre-OST0001 complete [ 7747.578131] LustreError: 333935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.579962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.583254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7747.598667] Lustre: Skipped 1 previous similar message [ 7752.538655] Lustre: server umount lustre-OST0000 complete [ 7757.793257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7757.796772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.802095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7757.802822] Lustre: Skipped 1 previous similar message [ 7757.806710] Lustre: Skipped 2 previous similar messages [ 7760.095863] Lustre: server umount lustre-MDT0000 complete [ 7761.529492] LustreError: 334928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450357 with bad export cookie 2289217280258879372 [ 7761.533156] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7761.533434] LustreError: 334928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7761.674395] Lustre: server umount lustre-MDT0001 complete [ 7765.698721] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7766.987610] Key type lgssc unregistered [ 7767.121385] LNet: 336881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7767.124122] LNetError: 336881:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7767.130770] LNet: Removed LNI 192.168.201.120@tcp [ 7767.486245] Key type .llcrypt unregistered [ 7767.487689] Key type ._llcrypt unregistered [ 7799.431750] Key type ._llcrypt registered [ 7799.433287] Key type .llcrypt registered [ 7799.471743] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 7805.444653] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 7805.762705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7805.812071] alg: No test for adler32 (adler32-zlib) [ 7806.671731] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 7806.767316] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 7808.351143] Key type lgssc registered [ 7808.708052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7820.814545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7820.823487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.921310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7821.937455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7821.975782] Lustre: lustre-MDT0000: new disk, initializing [ 7822.002096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.009101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7823.534392] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7828.437760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.491449] Lustre: 341788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7828.521890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7828.524959] Lustre: Skipped 1 previous similar message [ 7828.564574] Lustre: lustre-MDT0001: new disk, initializing [ 7828.601467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7828.617340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7828.620883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7830.441531] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7835.045783] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7837.381951] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7840.194817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7840.325404] Lustre: lustre-OST0000: new disk, initializing [ 7840.330873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7840.335040] Lustre: 343753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7840.370559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7841.673821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7841.677952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7841.729268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7842.966950] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7847.868747] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7849.906922] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7850.027404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7851.771908] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7851.848031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7857.123057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7857.124109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7857.126676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7857.131345] Lustre: Skipped 1 previous similar message [ 7857.135697] Lustre: Skipped 1 previous similar message [ 7858.703095] Lustre: server umount lustre-OST0000 complete [ 7862.240609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.244517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7862.244785] Lustre: Skipped 1 previous similar message [ 7866.210497] Lustre: server umount lustre-MDT0000 complete [ 7867.827214] LustreError: 341780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450464 with bad export cookie 1237693915626960396 [ 7867.830371] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7867.832237] LustreError: 341780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7868.016169] Lustre: server umount lustre-MDT0001 complete [ 7871.483312] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:54:27 (1783450467) [ 7874.571763] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:54:30 (1783450470) [ 7878.428311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.622351] LustreError: 346507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.632092] LustreError: 346507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7878.653569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.705963] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7883.747150] LustreError: 346508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.471234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7884.657663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7886.420795] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7889.514492] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7891.828751] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7894.903474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.068336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7897.953280] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7901.246232] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7915.164717] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7915.930199] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7916.689509] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7917.583243] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7918.486321] Lustre: Failing over lustre-MDT0000 [ 7918.690286] Lustre: server umount lustre-MDT0000 complete [ 7919.586174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.588577] LustreError: 346503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.595942] Lustre: Skipped 1 previous similar message [ 7923.142193] LustreError: 346504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.147908] LustreError: 346504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7928.247831] LustreError: 346503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.256710] LustreError: 346503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7932.496639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.567807] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7932.708991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7932.740675] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7933.372646] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7934.473849] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7938.022737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7938.040707] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7938.065407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7940.286373] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7940.972837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7943.062259] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7943.824307] Lustre: Failing over lustre-MDT0000 [ 7944.055785] Lustre: server umount lustre-MDT0000 complete [ 7948.258430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.260313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7948.264793] Lustre: Skipped 1 previous similar message [ 7948.265468] LustreError: 346504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.269165] LustreError: Skipped 1 previous similar message [ 7948.280017] LustreError: 346504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7957.839817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.897574] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7958.047177] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7958.967561] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7959.569356] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7963.108479] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7963.110579] Lustre: Skipped 2 previous similar messages [ 7963.121067] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7963.139929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7965.252064] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7965.850131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7973.346475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7973.347510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7973.348452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7973.348456] Lustre: Skipped 1 previous similar message [ 7973.355732] Lustre: Skipped 4 previous similar messages [ 7974.565690] Lustre: server umount lustre-MDT0000 complete [ 7976.126905] LustreError: 346488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450572 with bad export cookie 1237693915626963105 [ 7976.127252] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7976.132673] LustreError: 346488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7976.287492] Lustre: server umount lustre-MDT0001 complete [ 7988.059893] Lustre: server umount lustre-OST0000 complete [ 7994.014694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7994.166740] LustreError: 352857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7994.173340] LustreError: 352857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7994.192526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.195068] Lustre: Skipped 1 previous similar message [ 7995.704621] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 7998.812663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.384597] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8002.960946] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8004.875494] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8007.429826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8009.513560] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8011.946541] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8018.853928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8024.034585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.036379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.038478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.042697] Lustre: Skipped 3 previous similar messages [ 8025.752691] Lustre: server umount lustre-OST0000 complete [ 8029.154155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8029.156806] Lustre: Skipped 1 previous similar message [ 8033.178857] Lustre: server umount lustre-MDT0000 complete [ 8034.272137] LustreError: 352857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8034.278697] LustreError: 352857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8034.483369] LustreError: 352839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450630 with bad export cookie 1237693915626963854 [ 8034.486096] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8034.487766] LustreError: 352839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8034.610990] Lustre: server umount lustre-MDT0001 complete [ 8038.335985] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8039.383190] Key type lgssc unregistered [ 8039.528407] LNet: 356410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8039.530597] LNetError: 356410:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8039.537260] LNet: Removed LNI 192.168.201.120@tcp [ 8039.843924] Key type .llcrypt unregistered [ 8039.845294] Key type ._llcrypt unregistered [ 8049.968628] Key type ._llcrypt registered [ 8049.969988] Key type .llcrypt registered [ 8050.013142] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8050.450722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8050.463166] alg: No test for adler32 (adler32-zlib) [ 8051.325291] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8051.425430] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8053.023091] Key type lgssc registered [ 8053.442073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8056.518948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8056.527650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8057.701219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8059.259850] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8062.289144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.414483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8063.865376] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8066.460388] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8068.302719] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8070.618693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8070.751920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.737980] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8075.250201] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8082.026199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8092.642940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8092.644211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.648394] Lustre: Skipped 2 previous similar messages [ 8092.651336] Lustre: Skipped 2 previous similar messages [ 8096.158546] Lustre: server umount lustre-MDT0000 complete [ 8097.442100] LustreError: 359757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450693 with bad export cookie 16155043068901097252 [ 8097.445417] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8097.446511] LustreError: 359757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8097.560808] Lustre: server umount lustre-MDT0001 complete [ 8109.150140] Lustre: server umount lustre-OST0000 complete [ 8113.796145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.949875] LustreError: 361646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8113.957393] LustreError: 361646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8113.974315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.311195] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8118.275691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.416376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8120.049921] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8123.090553] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8125.121982] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8127.651063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8127.774322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.729797] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8132.221033] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8140.070238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8150.496331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.499418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.500559] Lustre: Skipped 2 previous similar messages [ 8150.505484] Lustre: Skipped 2 previous similar messages [ 8155.616618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.619058] Lustre: Skipped 2 previous similar messages [ 8155.926352] Lustre: server umount lustre-MDT0000 complete [ 8157.192149] LustreError: 361628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450753 with bad export cookie 16155043068901098358 [ 8157.195228] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.197016] LustreError: 361628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8157.308859] Lustre: server umount lustre-MDT0001 complete [ 8167.850587] Lustre: server umount lustre-OST0000 complete [ 8174.680753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.889476] LustreError: 365575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.924981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8176.892408] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8180.193470] LustreError: 365576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.513700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.169676] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8184.900200] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8187.134595] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8189.653911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.122752] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8194.685837] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8200.040511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8205.281281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8205.282695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.284275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8205.292608] Lustre: Skipped 1 previous similar message [ 8207.454351] Lustre: server umount lustre-OST0000 complete [ 8210.400732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8210.401425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8210.405475] Lustre: Skipped 1 previous similar message [ 8210.408190] Lustre: Skipped 1 previous similar message [ 8215.224321] Lustre: server umount lustre-MDT0000 complete [ 8215.520097] LustreError: 365576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.528595] LustreError: 365576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8216.608147] LustreError: 365557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450812 with bad export cookie 16155043068901099366 [ 8216.611018] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.612099] LustreError: 365557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8216.751116] Lustre: server umount lustre-MDT0001 complete [ 8220.776936] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8221.915231] Key type lgssc unregistered [ 8222.042359] LNet: 369103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8222.045152] LNetError: 369103:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8222.055185] LNet: Removed LNI 192.168.201.120@tcp [ 8222.386978] Key type .llcrypt unregistered [ 8222.388344] Key type ._llcrypt unregistered [ 8232.536174] Key type ._llcrypt registered [ 8232.537182] Key type .llcrypt registered [ 8232.574995] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8232.889786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8232.904546] alg: No test for adler32 (adler32-zlib) [ 8233.764384] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8233.860868] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8235.447373] Key type lgssc registered [ 8235.839707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8238.616584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8238.624138] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.626717] LustreError: 370520:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8238.637550] LustreError: 370520:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8238.640236] LustreError: 370520:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8238.643035] LustreError: 370520:0:(tgt_mount.c:2580:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8238.646287] LustreError: 370520:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8241.588475] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.591631] LustreError: 370917:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8241.600979] LustreError: 370917:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8241.603711] LustreError: 370917:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8241.606707] LustreError: 370917:0:(tgt_mount.c:2580:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8241.609922] LustreError: 370917:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8248.543143] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8249.668925] Key type lgssc unregistered [ 8249.813680] LNet: 371918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8249.820758] LNetError: 371918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8249.830107] LNet: Removed LNI 192.168.201.120@tcp [ 8250.185415] Key type .llcrypt unregistered [ 8250.186870] Key type ._llcrypt unregistered [ 8257.021032] Key type ._llcrypt registered [ 8257.022382] Key type .llcrypt registered [ 8257.068616] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8263.347038] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8263.674173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8263.708467] alg: No test for adler32 (adler32-zlib) [ 8264.575263] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8264.675920] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8266.263110] Key type lgssc registered [ 8266.634035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8279.151672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8279.160550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8280.263836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8280.278056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8280.325340] Lustre: lustre-MDT0000: new disk, initializing [ 8280.354372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.362319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8281.830482] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8286.317583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.367374] Lustre: 375060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8286.402084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8286.404245] Lustre: Skipped 1 previous similar message [ 8286.439454] Lustre: lustre-MDT0001: new disk, initializing [ 8286.481828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8286.498504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8286.502104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8287.927274] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8291.835762] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8293.821396] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8296.173811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8296.322175] Lustre: lustre-OST0000: new disk, initializing [ 8296.324269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8296.327591] Lustre: 377025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8296.360929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.456889] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8299.641298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8299.645784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8299.677306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8302.196810] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8303.631094] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8303.703757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8305.077817] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8305.167311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8310.241587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8310.241817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.245130] LustreError: Skipped 1 previous similar message [ 8310.252073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.254424] Lustre: Skipped 1 previous similar message [ 8311.964104] Lustre: server umount lustre-OST0000 complete [ 8315.361620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.366679] Lustre: Skipped 1 previous similar message [ 8315.369680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.372142] Lustre: Skipped 1 previous similar message [ 8319.395127] Lustre: server umount lustre-MDT0000 complete [ 8320.666122] LustreError: 375852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450916 with bad export cookie 18306833368247429640 [ 8320.669485] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.669775] LustreError: 375852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8320.788603] Lustre: server umount lustre-MDT0001 complete [ 8323.571434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8324.154232] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:01:59 (1783450919) [ 8326.424659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.576273] LustreError: 379479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8326.583523] LustreError: 379479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8326.606785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8328.026335] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8331.048135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.189131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8332.692816] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8335.346198] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8337.227164] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8339.582945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8339.713699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.708478] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8344.136322] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8346.648455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8346.720465] Lustre: lustre-OST0001: new disk, initializing [ 8346.722619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8346.725808] Lustre: 382313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8348.866209] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8351.239137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8351.243854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8351.275194] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8352.709601] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8356.326743] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8356.330092] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.335564] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8356.338693] Lustre: Skipped 1 previous similar message [ 8359.361833] Lustre: lustre-OST0001: Not available for connect from 192.168.201.20@tcp (stopping) [ 8360.701588] Lustre: server umount lustre-OST0001 complete [ 8361.443108] LustreError: 381297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.447538] LustreError: 381297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8366.562966] LustreError: 381298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.568371] LustreError: 381298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8371.683811] LustreError: 381482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.803844] LustreError: 381298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.805179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8376.806109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8376.806122] Lustre: Skipped 1 previous similar message [ 8376.807713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8376.813357] LustreError: 381298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8376.816637] LustreError: Skipped 2 previous similar messages [ 8378.967351] Lustre: server umount lustre-OST0000 complete [ 8381.920630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8381.924427] Lustre: Skipped 2 previous similar messages [ 8386.517734] Lustre: server umount lustre-MDT0000 complete [ 8387.040103] LustreError: 380205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.045683] LustreError: 380205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8387.795120] LustreError: 381320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783450984 with bad export cookie 18306833368247430522 [ 8387.798116] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8387.799420] LustreError: 381320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8402.399134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8402.496655] Lustre: server umount lustre-MDT0001 complete [ 8406.420788] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8407.515431] Key type lgssc unregistered [ 8407.643439] LNet: 384297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8407.647753] LNetError: 384297:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8407.658304] LNet: Removed LNI 192.168.201.120@tcp [ 8408.051223] Key type .llcrypt unregistered [ 8408.052801] Key type ._llcrypt unregistered [ 8427.551403] Key type ._llcrypt registered [ 8427.552492] Key type .llcrypt registered [ 8427.597751] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8427.973767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8428.001744] alg: No test for adler32 (adler32-zlib) [ 8428.860768] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8428.952117] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8430.535135] Key type lgssc registered [ 8430.890463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8433.479301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8433.487550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.571952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8434.578550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8434.633413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8436.017796] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8439.106858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.145227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8439.158112] Lustre: 387072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8439.262548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8440.747815] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8443.297304] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8445.300335] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8446.944943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.946032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8446.949867] Lustre: Skipped 1 previous similar message [ 8452.064744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.067371] Lustre: Skipped 2 previous similar messages [ 8457.184333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8457.186312] Lustre: Skipped 1 previous similar message [ 8460.255547] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8460.377139] Lustre: server umount lustre-MDT0000 complete [ 8461.867044] LustreError: 387812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451058 with bad export cookie 10960596847065707462 [ 8461.870227] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8461.871672] LustreError: 387812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8461.993961] Lustre: server umount lustre-MDT0001 complete [ 8465.719295] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:04:21 (1783451061) [ 8468.332203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.526391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.927316] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8473.051133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8473.192812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8474.591877] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8477.463243] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8479.320308] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8481.905037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8481.982845] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8482.048114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.155333] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8486.866548] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8490.660418] Lustre: Permanently deactivating lustre-OST0000 [ 8490.669630] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8492.720449] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8493.879329] LustreError: 392470:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8493.883355] LustreError: 392470:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8493.886025] LustreError: 392470:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8494.574894] LustreError: 392518:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8494.577379] LustreError: 392518:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8501.402717] Lustre: server umount lustre-OST0000 complete [ 8502.387469] LustreError: 392767:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8502.389751] LustreError: 392767:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8503.775863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8503.778893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.783902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8505.824666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.943952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.946650] Lustre: Skipped 3 previous similar messages [ 8513.013589] Lustre: server umount lustre-MDT0000 complete [ 8514.303216] LustreError: 389563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451110 with bad export cookie 10960596847065708029 [ 8514.306304] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8514.307629] LustreError: 389563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8523.788746] Lustre: server umount lustre-MDT0001 complete [ 8526.604192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.912930] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8528.799992] LustreError: 393924:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8530.825530] Lustre: 394263:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.120@tcp [ 8531.510731] Lustre: server umount MGS complete [ 8535.044956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.181351] LustreError: 394986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.185625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8535.187436] LustreError: 394986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8535.190341] Lustre: Skipped 1 previous similar message [ 8535.213218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8536.593641] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8539.293837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.375152] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8540.655576] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8542.910723] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8544.512379] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8545.083315] Lustre: Permanently reactivating lustre-OST0000 [ 8545.087472] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8545.089597] Lustre: Skipped 2 previous similar messages [ 8565.215572] LustreError: 396564:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8568.458366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8568.582245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8568.587758] Lustre: Skipped 1 previous similar message [ 8570.647658] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8573.277031] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8585.186161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8585.186611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8585.195180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.196739] Lustre: Skipped 1 previous similar message [ 8591.057252] Lustre: server umount lustre-OST0000 complete [ 8595.425028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.430384] Lustre: Skipped 1 previous similar message [ 8598.426515] Lustre: server umount lustre-MDT0000 complete [ 8599.649459] LustreError: 394967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451195 with bad export cookie 10960596847065709065 [ 8599.653040] LustreError: 394967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8599.654322] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8599.769341] Lustre: server umount lustre-MDT0001 complete [ 8603.376883] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8604.385702] Key type lgssc unregistered [ 8604.517405] LNet: 399464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8604.520654] LNetError: 399464:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8604.527296] LNet: Removed LNI 192.168.201.120@tcp [ 8604.826605] Key type .llcrypt unregistered [ 8604.828543] Key type ._llcrypt unregistered [ 8610.739410] Key type ._llcrypt registered [ 8610.740823] Key type .llcrypt registered [ 8610.773938] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8617.036732] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8617.372503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8617.404078] alg: No test for adler32 (adler32-zlib) [ 8618.251317] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8618.336659] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8619.919417] Key type lgssc registered [ 8620.278412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8631.222398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8631.227496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.308840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8632.319171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8632.354982] Lustre: lustre-MDT0000: new disk, initializing [ 8632.376845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.384879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8633.740924] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8637.873968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.914401] Lustre: 402593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8637.938170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8637.939934] Lustre: Skipped 1 previous similar message [ 8637.974790] Lustre: lustre-MDT0001: new disk, initializing [ 8638.003779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8638.015275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8638.019211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8639.309299] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8643.086447] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8644.851250] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8647.150936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8647.256641] Lustre: lustre-OST0000: new disk, initializing [ 8647.258775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8647.262435] Lustre: 404560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8647.292561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8649.185270] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8649.717234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8649.720299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8649.755482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8652.686125] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8654.047734] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8654.117245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8655.412188] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8655.484356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8659.937088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8659.938218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.939065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8659.945768] Lustre: Skipped 1 previous similar message [ 8662.235814] Lustre: server umount lustre-OST0000 complete [ 8663.521120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8663.524263] LustreError: Skipped 1 previous similar message [ 8663.525679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.529581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8663.531249] Lustre: Skipped 1 previous similar message [ 8665.056269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8665.056831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.656408] Lustre: server umount lustre-MDT0000 complete [ 8670.931714] LustreError: 402585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451267 with bad export cookie 9338365684527440076 [ 8670.934560] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.937062] LustreError: 402585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8671.064345] Lustre: server umount lustre-MDT0001 complete [ 8673.918552] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8674.510117] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:07:50 (1783451270) [ 8676.754722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.884445] LustreError: 407016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.890490] LustreError: 407016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8676.904744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.104504] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8680.845557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.959227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8682.216229] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8684.480001] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8686.240640] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8688.531032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8690.577961] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8693.001451] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8693.691652] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8704.520697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8707.041056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8707.042598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.044244] LustreError: Skipped 1 previous similar message [ 8707.044868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8707.049044] Lustre: Skipped 1 previous similar message [ 8707.055154] Lustre: Skipped 2 previous similar messages [ 8712.076393] Lustre: server umount lustre-OST0000 complete [ 8716.767722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8716.770780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8716.774416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8719.381739] Lustre: server umount lustre-MDT0000 complete [ 8720.602365] LustreError: 408857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451316 with bad export cookie 9338365684527440944 [ 8720.605542] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.608111] LustreError: 408857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8720.731715] Lustre: server umount lustre-MDT0001 complete [ 8724.411991] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8725.383790] Key type lgssc unregistered [ 8725.509415] LNet: 410760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8725.511731] LNetError: 410760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8725.519303] LNet: Removed LNI 192.168.201.120@tcp [ 8725.855698] Key type .llcrypt unregistered [ 8725.857089] Key type ._llcrypt unregistered [ 8733.247243] Key type ._llcrypt registered [ 8733.248678] Key type .llcrypt registered [ 8733.286493] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8733.370259] Key type .llcrypt unregistered [ 8733.371923] Key type ._llcrypt unregistered [ 8740.212102] Key type ._llcrypt registered [ 8740.213716] Key type .llcrypt registered [ 8740.251213] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8740.567191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8740.611382] alg: No test for adler32 (adler32-zlib) [ 8741.456269] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8741.543735] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8743.119150] Key type lgssc registered [ 8743.472881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8745.997779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8746.004502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.144421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.448186] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8751.108107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8751.215822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8753.100977] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8755.440266] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8756.705980] LustreError: 412947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.712798] LustreError: 412947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8757.669538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.730961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8757.733843] Lustre: Skipped 1 previous similar message [ 8757.735671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8757.807368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8759.141735] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8767.970816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8767.971792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.972582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8767.973551] LustreError: Skipped 1 previous similar message [ 8767.982360] Lustre: Skipped 1 previous similar message [ 8772.109245] Lustre: server umount lustre-OST0000 complete [ 8777.186433] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8777.189831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.196482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8777.198254] Lustre: Skipped 1 previous similar message [ 8778.208857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.606160] Lustre: server umount lustre-MDT0000 complete [ 8780.900100] LustreError: 415056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451377 with bad export cookie 13099857141056222176 [ 8780.902498] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8780.903817] LustreError: 415056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8781.040195] Lustre: server umount lustre-MDT0001 complete [ 8784.659926] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8785.602746] Key type lgssc unregistered [ 8785.728344] LNet: 416094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8785.731163] LNetError: 416094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8785.739244] LNet: Removed LNI 192.168.201.120@tcp [ 8786.048397] Key type .llcrypt unregistered [ 8786.049649] Key type ._llcrypt unregistered [ 8795.711215] Key type ._llcrypt registered [ 8795.712251] Key type .llcrypt registered [ 8795.751451] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8796.135467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8796.171738] alg: No test for adler32 (adler32-zlib) [ 8797.019526] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8797.105824] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8798.687154] Key type lgssc registered [ 8799.123875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8801.904537] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8801.916643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8817.375426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8822.495273] LustreError: 417460:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 8849.119177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8850.916867] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8853.243310] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8855.392413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.555698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8855.576969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8856.817143] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8859.555894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8859.694531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8860.994793] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8863.234046] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8864.933664] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8868.834238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.835443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8868.838934] Lustre: Skipped 1 previous similar message [ 8873.296311] Lustre: server umount lustre-OST0000 complete [ 8874.975684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8874.978313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8874.982334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8874.983923] Lustre: Skipped 1 previous similar message [ 8879.072171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8879.072434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8880.658465] Lustre: server umount lustre-MDT0000 complete [ 8881.823301] LustreError: 417466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451478 with bad export cookie 1061293301131024079 [ 8881.826568] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8881.827147] LustreError: 417466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8881.933671] Lustre: server umount lustre-MDT0001 complete [ 8885.443886] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8886.428809] Key type lgssc unregistered [ 8886.546295] LNet: 421028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8886.548542] LNetError: 421028:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8886.561272] LNet: Removed LNI 192.168.201.120@tcp [ 8886.873669] Key type .llcrypt unregistered [ 8886.874900] Key type ._llcrypt unregistered [ 8896.296484] Key type ._llcrypt registered [ 8896.298124] Key type .llcrypt registered [ 8896.335420] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8896.626235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8896.632847] alg: No test for adler32 (adler32-zlib) [ 8897.485943] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8897.579968] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8899.159106] Key type lgssc registered [ 8899.503721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8902.044345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8902.051855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.197161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.517378] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8907.222558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8907.341631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8908.614794] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8910.948164] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8912.696413] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8914.905780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8915.017811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.016338] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8919.422443] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8928.359523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8932.832218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8932.836383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.843474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8933.856803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.858175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8933.860564] Lustre: Skipped 1 previous similar message [ 8933.863618] Lustre: Skipped 2 previous similar messages [ 8936.726124] Lustre: server umount lustre-MDT0000 complete [ 8937.449465] LustreError: 422401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.20@tcp arrived at 1783451533 with bad export cookie 2036796980318749160 [ 8937.453345] LustreError: 422401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8938.185527] LustreError: 422447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451534 with bad export cookie 2036796980318748537 [ 8938.189797] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8938.190329] LustreError: 422447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8938.976307] LustreError: 422420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8938.977174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8938.981859] LustreError: 422420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8938.987055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8938.991727] Lustre: Skipped 1 previous similar message [ 8944.095883] LustreError: 422439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.096241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.102791] LustreError: 422439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8944.270668] Lustre: server umount lustre-OST0000 complete [ 8946.456414] Lustre: server umount lustre-MDT0001 complete [ 8949.886835] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 8950.849842] Key type lgssc unregistered [ 8950.979495] LNet: 426120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8950.982030] LNetError: 426120:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8950.988292] LNet: Removed LNI 192.168.201.120@tcp [ 8951.289620] Key type .llcrypt unregistered [ 8951.291031] Key type ._llcrypt unregistered [ 8960.255774] Key type ._llcrypt registered [ 8960.256975] Key type .llcrypt registered [ 8960.300841] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8960.806742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8960.827887] alg: No test for adler32 (adler32-zlib) [ 8961.689192] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 8961.772787] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8963.351125] Key type lgssc registered [ 8963.720433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8966.352114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8966.357851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.517744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.838096] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8971.679790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8971.818145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8973.181783] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8975.450668] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8977.307149] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8979.468698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8979.572473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.359076] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8983.691483] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8992.869335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8997.343871] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8997.347541] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8997.353742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8998.369746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8998.370881] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.293446] Lustre: server umount lustre-MDT0001 complete [ 9003.487963] LustreError: 427530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9003.488932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9003.489172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.489625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9003.489628] Lustre: Skipped 2 previous similar messages [ 9003.492274] LustreError: 427530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9008.608605] LustreError: 427511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9008.609155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9008.916500] Lustre: server umount lustre-OST0000 complete [ 9010.323200] Lustre: server umount lustre-MDT0000 complete [ 9014.691500] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9015.617826] Key type lgssc unregistered [ 9015.743357] LNet: 431216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9015.745613] LNetError: 431216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9015.753274] LNet: Removed LNI 192.168.201.120@tcp [ 9016.020752] Key type .llcrypt unregistered [ 9016.022660] Key type ._llcrypt unregistered [ 9023.664210] Key type ._llcrypt registered [ 9023.665454] Key type .llcrypt registered [ 9023.700891] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 9029.869905] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9030.242410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9030.254604] alg: No test for adler32 (adler32-zlib) [ 9031.091554] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 9031.171769] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9032.751161] Key type lgssc registered [ 9033.112658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9044.604902] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9049.655628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9049.661569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.739988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9050.748238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9050.780359] Lustre: lustre-MDT0000: new disk, initializing [ 9050.798737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.803594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9052.048900] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9057.631584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9057.721907] Lustre: lustre-OST0000: new disk, initializing [ 9057.723413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9057.724753] Lustre: Skipped 1 previous similar message [ 9057.726371] Lustre: 436221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9057.745894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9059.133842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9059.139233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9059.150054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9059.687614] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9063.315924] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9068.881500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9068.937468] Lustre: 435202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9068.972354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9069.009314] Lustre: lustre-MDT0001: new disk, initializing [ 9069.061611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9069.077658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9069.081066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9070.328143] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9073.832315] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9073.902661] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9075.204451] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9077.301436] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 9082.339075] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9082.339522] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9082.342515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9082.346835] Lustre: Skipped 1 previous similar message [ 9083.326076] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.20@tcp (stopping) [ 9083.329273] Lustre: Skipped 1 previous similar message [ 9087.457082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9087.459359] Lustre: Skipped 1 previous similar message [ 9092.578180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9092.580505] Lustre: Skipped 2 previous similar messages [ 9093.087114] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9093.143431] Lustre: server umount lustre-MDT0001 complete [ 9102.816596] LustreError: 438803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9102.822450] LustreError: 438803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9105.375847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9105.377934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9105.382230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9107.937200] LustreError: 435212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9111.446653] Lustre: server umount lustre-OST0000 complete [ 9126.879085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9126.961276] Lustre: server umount lustre-MDT0000 complete [ 9131.256266] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9132.208081] Key type lgssc unregistered [ 9132.331348] LNet: 439977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9132.334218] LNetError: 439977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9132.343225] LNet: Removed LNI 192.168.201.120@tcp [ 9132.638586] Key type .llcrypt unregistered [ 9132.639740] Key type ._llcrypt unregistered [ 9144.648913] Key type ._llcrypt registered [ 9144.649889] Key type .llcrypt registered [ 9144.689747] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9144.786282] Key type .llcrypt unregistered [ 9144.787296] Key type ._llcrypt unregistered [ 9151.071167] Key type ._llcrypt registered [ 9151.072036] Key type .llcrypt registered [ 9151.106044] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9151.398642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9151.448584] alg: No test for adler32 (adler32-zlib) [ 9152.289933] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [ 9152.372668] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9153.943078] Key type lgssc registered [ 9154.280788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9162.377075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9162.381289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.452261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9163.461819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9163.495395] Lustre: lustre-MDT0000: new disk, initializing [ 9163.515593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9163.520922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9164.723069] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9168.707471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.752347] Lustre: 443326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9168.777941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9168.779977] Lustre: Skipped 1 previous similar message [ 9168.816447] Lustre: lustre-MDT0001: new disk, initializing [ 9168.850894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9168.863547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9168.867230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9170.202160] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9174.044800] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9175.986646] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9178.381643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9178.493858] Lustre: lustre-OST0000: new disk, initializing [ 9178.496534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9178.500775] Lustre: 445292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9178.532803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.835072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9179.839746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9179.881605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9180.594104] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9184.413213] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9189.343907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9189.347492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9189.353917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9190.369533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9190.370197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9190.375590] Lustre: Skipped 1 previous similar message [ 9190.377950] Lustre: Skipped 1 previous similar message [ 9192.929892] Lustre: server umount lustre-MDT0000 complete [ 9194.265184] LustreError: 444120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451790 with bad export cookie 14299558531559707332 [ 9194.269044] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9194.270341] LustreError: 444120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9194.385611] Lustre: server umount lustre-MDT0001 complete [ 9204.886351] Lustre: server umount lustre-OST0000 complete [ 9208.285767] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:16:44 (1783451804) [ 9210.728891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9210.893596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.210922] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9214.817891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.941912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9216.259616] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9218.458806] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9220.123587] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9235.297541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9235.356314] Lustre: Found index 0 for lustre-OST0000, updating log [ 9235.412256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9236.453211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9237.118143] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9239.250722] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9241.313449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.313450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9241.319936] Lustre: Skipped 2 previous similar messages [ 9242.384573] Lustre: server umount lustre-OST0000 complete [ 9255.903983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9255.906405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.910631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9262.048730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9262.051552] Lustre: Skipped 3 previous similar messages [ 9269.215136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9269.271897] Lustre: server umount lustre-MDT0000 complete [ 9270.565278] LustreError: 447296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783451866 with bad export cookie 14299558531559708711 [ 9270.569500] LustreError: 447296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9270.570481] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9270.696339] Lustre: server umount lustre-MDT0001 complete [ 9273.513444] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:17:49 (1783451869) [ 9301.855296] LNet: 1 local NIs in recovery (showing 1): 192.168.201.120@tcp [ 9853.144579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.289237] LustreError: 453713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.298552] LustreError: 453713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9853.313471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9854.636932] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9857.444705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9858.928400] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9861.341376] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9863.198047] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9878.562354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9878.653708] Lustre: Found index 0 for lustre-OST0000, updating log [ 9880.727666] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9880.809465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9883.238538] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9884.897147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.897920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.898654] Lustre: Skipped 4 previous similar messages [ 9884.901608] Lustre: Skipped 1 previous similar message [ 9885.920775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9887.969782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9887.971637] Lustre: Skipped 1 previous similar message [ 9889.995486] Lustre: server umount lustre-OST0000 complete [ 9892.257084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.258575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9892.262145] Lustre: Skipped 1 previous similar message [ 9892.265584] Lustre: Skipped 1 previous similar message [ 9897.364665] Lustre: server umount lustre-MDT0000 complete [ 9897.376237] LustreError: 453713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.383207] LustreError: 453713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9898.400259] LustreError: 453714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.404728] LustreError: 453714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9898.609141] LustreError: 455610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783452494 with bad export cookie 14299558531559709768 [ 9898.612383] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.612947] LustreError: 455610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9898.727395] Lustre: server umount lustre-MDT0001 complete [ 9915.231205] LNet: 1 local NIs in recovery (showing 1): 192.168.201.120@tcp [ 9916.097815] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 9919.492204] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9931.263272] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:28:47 (1783452527) [ 9937.295714] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9938.401849] Key type lgssc unregistered [ 9940.575429] LNet: 460361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9940.579160] LNetError: 460361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10008.165280] LNet: Removed LNI 192.168.201.120@tcp [10008.469280] Key type .llcrypt unregistered [10008.470273] Key type ._llcrypt unregistered [10015.269434] Key type ._llcrypt registered [10015.270484] Key type .llcrypt registered [10015.305390] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10015.645985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10015.685941] alg: No test for adler32 (adler32-zlib) [10016.529055] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [10016.612646] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10018.199210] Key type lgssc registered [10018.559771] Lustre: Echo OBD driver; http://www.lustre.org/ [10019.998363] LNet: Added LNI 192.168.201.120@tcp100 [8/256/0/180] [10020.293229] LNet: Added LNI 192.168.201.120@tcp101 [8/256/0/180] [10020.577342] LNet: Added LNI 192.168.201.120@tcp102 [8/256/0/180] [10020.862523] LNet: Added LNI 192.168.201.120@tcp103 [8/256/0/180] [10021.159297] LNet: Added LNI 192.168.201.120@tcp104 [8/256/0/180] [10021.461840] LNet: Added LNI 192.168.201.120@tcp105 [8/256/0/180] [10021.770588] LNet: Added LNI 192.168.201.120@tcp106 [8/256/0/180] [10022.074401] LNet: Added LNI 192.168.201.120@tcp107 [8/256/0/180] [10022.374905] LNet: Added LNI 192.168.201.120@tcp108 [8/256/0/180] [10022.681412] LNet: Added LNI 192.168.201.120@tcp109 [8/256/0/180] [10022.984126] LNet: Added LNI 192.168.201.120@tcp110 [8/256/0/180] [10023.288619] LNet: Added LNI 192.168.201.120@tcp111 [8/256/0/180] [10023.584122] LNet: Added LNI 192.168.201.120@tcp112 [8/256/0/180] [10023.882670] LNet: Added LNI 192.168.201.120@tcp113 [8/256/0/180] [10024.164620] LNet: Added LNI 192.168.201.120@tcp114 [8/256/0/180] [10024.450298] LNet: Added LNI 192.168.201.120@tcp115 [8/256/0/180] [10024.736638] LNet: Added LNI 192.168.201.120@tcp116 [8/256/0/180] [10025.027987] LNet: Added LNI 192.168.201.120@tcp117 [8/256/0/180] [10025.306172] LNet: Added LNI 192.168.201.120@tcp118 [8/256/0/180] [10025.590603] LNet: Added LNI 192.168.201.120@tcp119 [8/256/0/180] [10025.891933] LNet: Added LNI 192.168.201.120@tcp120 [8/256/0/180] [10026.194755] LNet: Added LNI 192.168.201.120@tcp121 [8/256/0/180] [10026.500274] LNet: Added LNI 192.168.201.120@tcp122 [8/256/0/180] [10026.804029] LNet: Added LNI 192.168.201.120@tcp123 [8/256/0/180] [10027.121749] LNet: Added LNI 192.168.201.120@tcp124 [8/256/0/180] [10027.447673] LNet: Added LNI 192.168.201.120@tcp125 [8/256/0/180] [10027.739263] LNet: Added LNI 192.168.201.120@tcp126 [8/256/0/180] [10028.048503] LNet: Added LNI 192.168.201.120@tcp127 [8/256/0/180] [10028.362319] LNet: Added LNI 192.168.201.120@tcp128 [8/256/0/180] [10028.676791] LNet: Added LNI 192.168.201.120@tcp129 [8/256/0/180] [10028.976455] LNet: Added LNI 192.168.201.120@tcp130 [8/256/0/180] [10029.276139] LNet: Added LNI 192.168.201.120@tcp131 [8/256/0/180] [10029.556141] LNet: Added LNI 192.168.201.120@tcp132 [8/256/0/180] [10029.844574] LNet: Added LNI 192.168.201.120@tcp133 [8/256/0/180] [10030.130154] LNet: Added LNI 192.168.201.120@tcp134 [8/256/0/180] [10030.424104] LNet: Added LNI 192.168.201.120@tcp135 [8/256/0/180] [10032.696597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10032.700991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.736345] Lustre: 463358:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.120@tcp: can't add NID 192.168.201.120@tcp130: rc = -75 [10033.796684] LNetError: 463390:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.120 [10033.812951] LNetError: Protocol error connecting to 192.168.201.120@tcp124 at host 192.168.201.120:988: is it running a compatible version of Lustre? [10033.818650] LNetError: Skipped 2 previous similar messages [10033.851871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10033.864639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10033.903423] Lustre: lustre-MDT0000: new disk, initializing [10033.930244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.936835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10034.015148] LNet: 2 local NIs in recovery (showing 2): 192.168.201.120@tcp129, 192.168.201.120@tcp124 [10035.243355] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10036.064389] LNetError: 460823:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.120 [10036.069107] LNetError: 460823:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [10036.072792] LNetError: Protocol error connecting to 192.168.201.120@tcp129 at host 192.168.201.120:988: is it running a compatible version of Lustre? [10036.077714] LNetError: Skipped 13 previous similar messages [10039.323511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.378544] Lustre: 463375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10039.403412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10039.405100] Lustre: Skipped 1 previous similar message [10039.445971] Lustre: lustre-MDT0001: new disk, initializing [10039.459534] Lustre: 464181:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.120@tcp: can't add NID 192.168.201.120@tcp130: rc = -75 [10039.462365] Lustre: 464181:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10039.507901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10039.519473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10039.522476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10040.160153] LNetError: 460824:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.120 [10040.163937] LNetError: Protocol error connecting to 192.168.201.120@tcp129 at host 192.168.201.120:988: is it running a compatible version of Lustre? [10040.861650] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10043.889219] Lustre: 464750:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.120@tcp: can't add NID 192.168.201.120@tcp130: rc = -75 [10043.893850] Lustre: 464750:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10044.557071] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10046.384541] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10048.353413] LNetError: 463391:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.120 [10048.356960] LNetError: Protocol error connecting to 192.168.201.120@tcp129 at host 192.168.201.120:988: is it running a compatible version of Lustre? [10048.647716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.765097] Lustre: lustre-OST0000: new disk, initializing [10048.766985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10048.769250] Lustre: 465352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10048.803818] Lustre: 465357:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.120@tcp: can't add NID 192.168.201.120@tcp130: rc = -75 [10048.806308] Lustre: 465357:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [10048.823988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10049.987629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10049.993132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10050.042415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10050.745135] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10054.399751] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10060.258252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.258594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10060.262927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10060.264017] LustreError: Skipped 1 previous similar message [10060.266992] Lustre: Skipped 1 previous similar message [10062.798910] Lustre: server umount lustre-OST0000 complete [10063.713465] LNetError: 463392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.120 [10063.716696] LNetError: Protocol error connecting to 192.168.201.120@tcp126 at host 192.168.201.120:988: is it running a compatible version of Lustre? [10065.377872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10065.377985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.379184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10065.388836] Lustre: Skipped 2 previous similar messages [10070.175213] Lustre: server umount lustre-MDT0000 complete [10071.423532] LustreError: 463366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783452667 with bad export cookie 11663580630455575268 [10071.427542] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10071.427766] LustreError: 463366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10071.550064] Lustre: server umount lustre-MDT0001 complete [10071.906444] LNetError: 463390:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.120 [10071.910201] LNetError: 463390:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [10071.912796] LNetError: Protocol error connecting to 192.168.201.120@tcp112 at host 192.168.201.120:988: is it running a compatible version of Lustre? [10071.916387] LNetError: Skipped 5 previous similar messages [10075.393841] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10076.464380] Key type lgssc unregistered [10076.624769] LNet: 467263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10076.628944] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10076.631637] LNet: Removed LNI 192.168.201.120@tcp [10077.663171] LNet: Removed LNI 192.168.201.120@tcp100 [10077.664557] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10077.666963] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10077.671307] LNet: Removed LNI 192.168.201.120@tcp101 [10078.687190] LNet: Removed LNI 192.168.201.120@tcp102 [10078.688544] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10078.690570] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10078.693489] LNet: Removed LNI 192.168.201.120@tcp103 [10079.711186] LNet: Removed LNI 192.168.201.120@tcp104 [10080.735178] LNet: Removed LNI 192.168.201.120@tcp105 [10080.736799] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10080.739922] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10080.742675] LNet: Removed LNI 192.168.201.120@tcp106 [10081.759139] LNet: Removed LNI 192.168.201.120@tcp107 [10082.783173] LNet: Removed LNI 192.168.201.120@tcp108 [10083.807154] LNet: Removed LNI 192.168.201.120@tcp109 [10084.831142] LNet: Removed LNI 192.168.201.120@tcp110 [10084.832576] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10084.834769] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10084.837020] LNet: Removed LNI 192.168.201.120@tcp111 [10085.855133] LNet: Removed LNI 192.168.201.120@tcp112 [10086.879174] LNet: Removed LNI 192.168.201.120@tcp113 [10087.903165] LNet: Removed LNI 192.168.201.120@tcp114 [10088.927143] LNet: Removed LNI 192.168.201.120@tcp115 [10089.951146] LNet: Removed LNI 192.168.201.120@tcp116 [10090.975149] LNet: Removed LNI 192.168.201.120@tcp117 [10091.999140] LNet: Removed LNI 192.168.201.120@tcp118 [10093.023143] LNet: Removed LNI 192.168.201.120@tcp119 [10093.024683] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10093.027202] LNetError: 467263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10093.030043] LNet: Removed LNI 192.168.201.120@tcp120 [10094.047320] LNet: Removed LNI 192.168.201.120@tcp121 [10095.071146] LNet: Removed LNI 192.168.201.120@tcp122 [10096.095167] LNet: Removed LNI 192.168.201.120@tcp123 [10097.119134] LNet: Removed LNI 192.168.201.120@tcp124 [10098.143171] LNet: Removed LNI 192.168.201.120@tcp125 [10099.167140] LNet: Removed LNI 192.168.201.120@tcp126 [10100.191210] LNet: Removed LNI 192.168.201.120@tcp127 [10101.215151] LNet: Removed LNI 192.168.201.120@tcp128 [10101.216769] LNet: Removed LNI 192.168.201.120@tcp129 [10101.218089] LNet: Removed LNI 192.168.201.120@tcp130 [10101.219501] LNet: Removed LNI 192.168.201.120@tcp131 [10101.220901] LNet: Removed LNI 192.168.201.120@tcp132 [10101.222075] LNet: Removed LNI 192.168.201.120@tcp133 [10101.223352] LNet: Removed LNI 192.168.201.120@tcp134 [10101.231621] LNet: Removed LNI 192.168.201.120@tcp135 [10101.537143] Key type .llcrypt unregistered [10101.538195] Key type ._llcrypt unregistered [10109.058229] Key type ._llcrypt registered [10109.059196] Key type .llcrypt registered [10109.103599] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10115.357654] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10115.948191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10115.957747] alg: No test for adler32 (adler32-zlib) [10116.822028] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [10116.916454] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10118.495092] Key type lgssc registered [10118.906762] Lustre: Echo OBD driver; http://www.lustre.org/ [10130.588419] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:32:06 (1783452726) [10136.525883] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10137.577947] Key type lgssc unregistered [10137.705354] LNet: 471320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10137.708235] LNetError: 471320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10137.716267] LNet: Removed LNI 192.168.201.120@tcp [10138.013361] Key type .llcrypt unregistered [10138.014589] Key type ._llcrypt unregistered [10144.926989] Key type ._llcrypt registered [10144.928640] Key type .llcrypt registered [10144.972086] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10145.461680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10145.519787] alg: No test for adler32 (adler32-zlib) [10146.389573] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [10146.497761] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10148.087157] Key type lgssc registered [10148.499988] Lustre: Echo OBD driver; http://www.lustre.org/ [10151.773223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10151.779707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10152.867078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10152.876701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10152.916529] Lustre: lustre-MDT0000: new disk, initializing [10152.937688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10152.943348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10154.388861] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10158.519162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.560447] Lustre: 472508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10158.586209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10158.588412] Lustre: Skipped 1 previous similar message [10158.628506] Lustre: lustre-MDT0001: new disk, initializing [10158.658780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10158.671438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10158.675051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10160.015279] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10163.771064] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10165.570848] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10167.743626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10167.844515] Lustre: lustre-OST0000: new disk, initializing [10167.846101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10167.848793] Lustre: 474474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10167.877535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10170.064887] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10173.932315] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10173.945359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10173.948354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10173.987091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10175.089077] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing lnet_if_list [10176.701880] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10176.711971] LNet: Added LNI 192.168.201.120@tcp10 [8/256/0/180] [10178.411508] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10178.420228] LNet: Added LNI 192.168.201.120@tcp11 [8/256/0/180] [10179.700479] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10179.713312] LNet: Added LNI 192.168.201.120@tcp12 [8/256/0/180] [10181.076699] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10181.085810] LNet: Added LNI 192.168.201.120@tcp13 [8/256/0/180] [10182.442286] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10182.452561] LNet: Added LNI 192.168.201.120@tcp14 [8/256/0/180] [10183.864640] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10183.875508] LNet: Added LNI 192.168.201.120@tcp15 [8/256/0/180] [10185.381365] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10185.394557] LNet: Added LNI 192.168.201.120@tcp16 [8/256/0/180] [10186.865870] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10186.880521] LNet: Added LNI 192.168.201.120@tcp17 [8/256/0/180] [10188.423704] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10188.433881] LNet: Added LNI 192.168.201.120@tcp18 [8/256/0/180] [10190.124989] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10190.137352] LNet: Added LNI 192.168.201.120@tcp19 [8/256/0/180] [10191.945329] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10191.956539] LNet: Added LNI 192.168.201.120@tcp20 [8/256/0/180] [10193.599770] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10193.609977] LNet: Added LNI 192.168.201.120@tcp21 [8/256/0/180] [10195.277933] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10195.287649] LNet: Added LNI 192.168.201.120@tcp22 [8/256/0/180] [10196.978284] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10196.990619] LNet: Added LNI 192.168.201.120@tcp23 [8/256/0/180] [10198.707691] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10198.722091] LNet: Added LNI 192.168.201.120@tcp24 [8/256/0/180] [10200.490318] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10200.502349] LNet: Added LNI 192.168.201.120@tcp25 [8/256/0/180] [10202.419865] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10202.434443] LNet: Added LNI 192.168.201.120@tcp26 [8/256/0/180] [10204.430869] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10204.442508] LNet: Added LNI 192.168.201.120@tcp27 [8/256/0/180] [10206.469170] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10206.478913] LNet: Added LNI 192.168.201.120@tcp28 [8/256/0/180] [10208.464285] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10208.475358] LNet: Added LNI 192.168.201.120@tcp29 [8/256/0/180] [10210.402309] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10210.413671] LNet: Added LNI 192.168.201.120@tcp30 [8/256/0/180] [10212.408605] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10212.419023] LNet: Added LNI 192.168.201.120@tcp31 [8/256/0/180] [10214.461171] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10214.472979] LNet: Added LNI 192.168.201.120@tcp32 [8/256/0/180] [10216.598727] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10216.612602] LNet: Added LNI 192.168.201.120@tcp33 [8/256/0/180] [10218.863623] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10218.876437] LNet: Added LNI 192.168.201.120@tcp34 [8/256/0/180] [10221.161873] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10221.173701] LNet: Added LNI 192.168.201.120@tcp35 [8/256/0/180] [10223.451205] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10223.461381] LNet: Added LNI 192.168.201.120@tcp36 [8/256/0/180] [10225.849177] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10225.860026] LNet: Added LNI 192.168.201.120@tcp37 [8/256/0/180] [10228.338572] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10228.354715] LNet: Added LNI 192.168.201.120@tcp38 [8/256/0/180] [10230.690576] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10230.700403] LNet: Added LNI 192.168.201.120@tcp39 [8/256/0/180] [10232.987109] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10233.001344] LNet: Added LNI 192.168.201.120@tcp40 [8/256/0/180] [10235.305400] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10235.316814] LNet: Added LNI 192.168.201.120@tcp41 [8/256/0/180] [10237.725319] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10237.738226] LNet: Added LNI 192.168.201.120@tcp42 [8/256/0/180] [10240.076782] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10240.090800] LNet: Added LNI 192.168.201.120@tcp43 [8/256/0/180] [10242.524678] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10242.535862] LNet: Added LNI 192.168.201.120@tcp44 [8/256/0/180] [10245.268813] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10245.285567] LNet: Added LNI 192.168.201.120@tcp45 [8/256/0/180] [10247.953133] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10247.965426] LNet: Added LNI 192.168.201.120@tcp46 [8/256/0/180] [10250.901888] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10250.915673] LNet: Added LNI 192.168.201.120@tcp47 [8/256/0/180] [10253.798961] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10253.811355] LNet: Added LNI 192.168.201.120@tcp48 [8/256/0/180] [10256.596447] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10256.609971] LNet: Added LNI 192.168.201.120@tcp49 [8/256/0/180] [10259.269708] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10259.281533] LNet: Added LNI 192.168.201.120@tcp50 [8/256/0/180] [10269.153716] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.154658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10269.158374] Lustre: Skipped 2 previous similar messages [10269.161344] Lustre: Skipped 1 previous similar message [10272.795171] Lustre: server umount lustre-MDT0000 complete [10274.070637] LustreError: 472501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783452870 with bad export cookie 1380492441371598998 [10274.075095] LustreError: 472501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10274.075697] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10274.205476] Lustre: server umount lustre-MDT0001 complete [10281.530321] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10284.485768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10290.655205] Lustre: 471784:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783452870/real 1783452870] req@00000000deb6bc4e x1870085742479104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783452886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10290.662452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.751162] Lustre: 471782:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783452875/real 1783452875] req@000000005a4cbede x1870085742479232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783452891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10298.847484] LustreError: 471781:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000afe1ad79 x1870085742480000/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10298.974870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10299.999136] Lustre: 471782:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783452880/real 1783452880] req@00000000de3a13eb x1870085742479360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783452896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10300.346996] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10303.287578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.439460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10304.911749] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10305.375148] Lustre: 471784:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783452885/real 1783452885] req@00000000dae0e359 x1870085742479616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783452901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10307.390175] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10308.578939] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10309.181868] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10309.604861] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10318.817917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10318.819988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.823883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.327944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10319.842187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.844352] Lustre: Skipped 1 previous similar message [10322.710906] Lustre: server umount lustre-OST0000 complete [10324.961428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.962605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10324.967135] Lustre: Skipped 2 previous similar messages [10324.972749] Lustre: Skipped 1 previous similar message [10330.080305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10330.082185] Lustre: Skipped 1 previous similar message [10338.271126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10338.342914] Lustre: server umount lustre-MDT0000 complete [10339.636235] LustreError: 472501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783452935 with bad export cookie 1380492441371601035 [10339.640126] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10339.640280] LustreError: 472501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10339.756325] Lustre: server umount lustre-MDT0001 complete [10343.528686] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10344.553579] Key type lgssc unregistered [10344.716498] LNet: 501441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10344.719557] LNetError: 501441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10344.721509] LNet: Removed LNI 192.168.201.120@tcp [10344.722637] LNet: Removed LNI 192.168.201.120@tcp10 [10344.723796] LNet: Removed LNI 192.168.201.120@tcp11 [10344.725070] LNet: Removed LNI 192.168.201.120@tcp12 [10344.726152] LNet: Removed LNI 192.168.201.120@tcp13 [10344.727274] LNet: Removed LNI 192.168.201.120@tcp14 [10344.728673] LNet: Removed LNI 192.168.201.120@tcp15 [10344.729802] LNet: Removed LNI 192.168.201.120@tcp16 [10344.730884] LNet: Removed LNI 192.168.201.120@tcp17 [10344.731948] LNet: Removed LNI 192.168.201.120@tcp18 [10344.733168] LNet: Removed LNI 192.168.201.120@tcp19 [10344.734228] LNet: Removed LNI 192.168.201.120@tcp20 [10344.735360] LNet: Removed LNI 192.168.201.120@tcp21 [10344.736444] LNet: Removed LNI 192.168.201.120@tcp22 [10344.737456] LNet: Removed LNI 192.168.201.120@tcp23 [10344.738455] LNet: Removed LNI 192.168.201.120@tcp24 [10344.739459] LNet: Removed LNI 192.168.201.120@tcp25 [10344.740463] LNet: Removed LNI 192.168.201.120@tcp26 [10344.741469] LNet: Removed LNI 192.168.201.120@tcp27 [10344.742470] LNet: Removed LNI 192.168.201.120@tcp28 [10344.743446] LNet: Removed LNI 192.168.201.120@tcp29 [10344.744412] LNet: Removed LNI 192.168.201.120@tcp30 [10344.745449] LNet: Removed LNI 192.168.201.120@tcp31 [10344.746478] LNet: Removed LNI 192.168.201.120@tcp32 [10344.747577] LNet: Removed LNI 192.168.201.120@tcp33 [10344.748582] LNet: Removed LNI 192.168.201.120@tcp34 [10344.749556] LNet: Removed LNI 192.168.201.120@tcp35 [10344.750553] LNet: Removed LNI 192.168.201.120@tcp36 [10344.751629] LNet: Removed LNI 192.168.201.120@tcp37 [10344.752656] LNet: Removed LNI 192.168.201.120@tcp38 [10344.753664] LNet: Removed LNI 192.168.201.120@tcp39 [10344.754781] LNet: Removed LNI 192.168.201.120@tcp40 [10344.755785] LNet: Removed LNI 192.168.201.120@tcp41 [10344.756754] LNet: Removed LNI 192.168.201.120@tcp42 [10344.757770] LNet: Removed LNI 192.168.201.120@tcp43 [10344.758733] LNet: Removed LNI 192.168.201.120@tcp44 [10344.759676] LNet: Removed LNI 192.168.201.120@tcp45 [10344.760623] LNet: Removed LNI 192.168.201.120@tcp46 [10344.761667] LNet: Removed LNI 192.168.201.120@tcp47 [10344.762644] LNet: Removed LNI 192.168.201.120@tcp48 [10344.763611] LNet: Removed LNI 192.168.201.120@tcp49 [10344.770321] LNet: Removed LNI 192.168.201.120@tcp50 [10345.050188] Key type .llcrypt unregistered [10345.051840] Key type ._llcrypt unregistered [10351.933940] Key type ._llcrypt registered [10351.934952] Key type .llcrypt registered [10351.969736] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10358.117972] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10358.539529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10358.565174] alg: No test for adler32 (adler32-zlib) [10359.414862] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [10359.513394] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10361.095123] Key type lgssc registered [10361.507501] Lustre: Echo OBD driver; http://www.lustre.org/ [10372.924305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10372.929383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10374.019820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10374.029495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10374.067244] Lustre: lustre-MDT0000: new disk, initializing [10374.089858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10374.095615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10375.450949] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10379.603755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.642715] Lustre: 504608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10379.666634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10379.668488] Lustre: Skipped 1 previous similar message [10379.704906] Lustre: lustre-MDT0001: new disk, initializing [10379.737820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10379.749890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10379.755060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10381.068078] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10384.851393] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10386.616171] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10389.214079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10389.298709] Lustre: *** cfs_fail_loc=90f, val=0*** [10389.334306] Lustre: lustre-OST0000: new disk, initializing [10389.335772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10389.338118] Lustre: 506626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10389.371694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10391.301434] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10394.948917] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10397.404738] Lustre: server umount lustre-OST0000 complete [10400.223912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10400.226563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10400.231353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10402.784455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10402.784934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10404.827211] Lustre: server umount lustre-MDT0000 complete [10406.092432] LustreError: 504602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783453002 with bad export cookie 3782246227280776965 [10406.095736] LustreError: 504602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10406.095738] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10406.221453] Lustre: server umount lustre-MDT0001 complete [10409.859762] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10410.869156] Key type lgssc unregistered [10410.996356] LNet: 508524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10410.998676] LNetError: 508524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10411.006278] LNet: Removed LNI 192.168.201.120@tcp [10411.317474] Key type .llcrypt unregistered [10411.318531] Key type ._llcrypt unregistered [10419.271916] Key type ._llcrypt registered [10419.272966] Key type .llcrypt registered [10419.314794] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10419.721759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10419.760534] alg: No test for adler32 (adler32-zlib) [10420.632761] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [10420.738772] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10422.327165] Key type lgssc registered [10422.792937] Lustre: Echo OBD driver; http://www.lustre.org/ [10425.578199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10425.586642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.749911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10428.123623] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10431.043457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10431.179089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10432.643562] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10435.039238] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10436.881519] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10439.296396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10439.439544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10441.422343] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10441.516250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10441.520214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10443.829303] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10451.764801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10458.148040] loop0: detected capacity change from 0 to 204800000 [10458.275437] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10458.279343] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10458.283248] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10460.363075] loop0: detected capacity change from 0 to 204800000 [10460.431475] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10460.434767] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10460.438482] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10462.228601] loop0: detected capacity change from 0 to 204800000 [10464.194742] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.245855] Lustre: fs15246-OST0000: new disk, initializing [10464.247652] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10464.251811] Lustre: 513383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10464.272785] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10466.318119] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10470.365585] loop1: detected capacity change from 0 to 204800000 [10472.256527] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10472.294177] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10472.320764] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10472.357067] Lustre: fs15246-MDT0000: new disk, initializing [10472.404556] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10472.414855] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10472.458651] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10473.503740] Lustre: 514502:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000dd82f798 x1870086030785920/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783453085 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10473.826545] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10482.591267] LustreError: 514502:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [10482.594459] LustreError: 514502:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [10482.597575] LustreError: 514502:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [10486.244341] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10486.248656] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10487.440300] Lustre: server umount fs15246-MDT0000 complete [10490.574930] Lustre: server umount fs15246-OST0000 complete [10496.480891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.483398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10496.484528] Lustre: Skipped 1 previous similar message [10500.251850] Lustre: server umount lustre-OST0000 complete [10501.600774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10501.601736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10501.604102] Lustre: Skipped 1 previous similar message [10501.607626] Lustre: Skipped 2 previous similar messages [10506.720495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.722763] Lustre: Skipped 1 previous similar message [10507.734218] Lustre: server umount lustre-MDT0000 complete [10508.983293] LustreError: 509796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783453105 with bad export cookie 4919197097676575124 [10508.985755] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10508.987079] LustreError: 509796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10509.099670] Lustre: server umount lustre-MDT0001 complete [10512.745467] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10513.760108] Key type lgssc unregistered [10513.885297] LNet: 517598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10513.887644] LNetError: 517598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10513.895353] LNet: Removed LNI 192.168.201.120@tcp [10514.231992] Key type .llcrypt unregistered [10514.232974] Key type ._llcrypt unregistered [10521.523132] Key type ._llcrypt registered [10521.524104] Key type .llcrypt registered [10521.568311] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10521.883116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10521.897699] alg: No test for adler32 (adler32-zlib) [10522.768551] Lustre: Lustre: Build Version: 2.17.54_105_g2b8c50a [10522.863289] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10524.447131] Key type lgssc registered [10524.837205] Lustre: Echo OBD driver; http://www.lustre.org/ [10537.594660] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10540.617609] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10552.863177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10552.871062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.965728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10553.976266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10554.013827] Lustre: lustre-MDT0000: new disk, initializing [10554.035079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10554.040191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10555.537061] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10559.910475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10559.958626] Lustre: 522040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10559.989428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10559.992532] Lustre: Skipped 1 previous similar message [10560.037065] Lustre: lustre-MDT0001: new disk, initializing [10560.076275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10560.091967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10560.095798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10561.575889] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10565.337468] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10567.158933] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10569.400031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10569.528570] Lustre: lustre-OST0000: new disk, initializing [10569.530263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10569.533000] Lustre: 524008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10569.574179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10570.560816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10570.564236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10570.606874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10571.696339] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10575.413041] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10576.813992] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10576.885889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10578.249300] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10578.320204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10580.962124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10580.963565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10580.964818] LustreError: Skipped 1 previous similar message [10580.972582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10580.975717] Lustre: Skipped 1 previous similar message [10586.083783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10586.086151] Lustre: Skipped 1 previous similar message [10591.201598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10591.203330] Lustre: Skipped 1 previous similar message [10593.247182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10593.340890] Lustre: server umount lustre-OST0000 complete [10595.808816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10595.814627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10595.822624] Lustre: Skipped 1 previous similar message [10595.826179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10601.134180] Lustre: server umount lustre-MDT0000 complete [10602.338242] LustreError: 522033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783453198 with bad export cookie 3047680103055865788 [10602.341532] LustreError: 522033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10602.341789] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10602.462333] Lustre: server umount lustre-MDT0001 complete [10611.872541] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10616.312387] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10627.623471] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10354 sec ========= 15:40:23 (1783453223) [10628.130970] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:40:23 (1783453223) === [10629.285921] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:40:25 (1783453225) === [10645.150602] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10646.580667] Key type lgssc unregistered [10646.732594] LNet: 530639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10646.735262] LNetError: 530639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10646.744629] LNet: Removed LNI 192.168.201.120@tcp [10647.158402] Key type .llcrypt unregistered [10647.160222] Key type ._llcrypt unregistered