[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 804718083 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003170] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008572] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011127] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.023124] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028424] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033015] *** VALIDATE tmpfs *** [ 0.034432] *** VALIDATE proc *** [ 0.035225] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037252] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040024] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044607] debug: unmapping init [mem 0xffffffffa9659000-0xffffffffa9660fff] [ 0.047240] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048787] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049020] ... version: 2 [ 0.050010] ... bit width: 48 [ 0.051008] ... generic registers: 4 [ 0.052010] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054009] ... fixed-purpose events: 3 [ 0.055009] ... event mask: 000000070000000f [ 0.057291] rcu: Hierarchical SRCU implementation. [ 0.059567] smp: Bringing up secondary CPUs ... [ 0.060538] x86: Booting SMP configuration: [ 0.061018] .... node #0, CPUs: #1 #2 #3 [ 0.067160] smp: Brought up 1 node, 4 CPUs [ 0.069013] smpboot: Max logical packages: 1 [ 0.070012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101948] node 0 deferred pages initialised in 29ms [ 0.105568] devtmpfs: initialized [ 0.106370] x86/mm: Memory block size: 128MB [ 0.109265] gcov: version magic: 0x41383552 [ 0.111437] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.112071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113350] pinctrl core: initialized pinctrl subsystem [ 0.114531] [ 0.115007] ************************************************************* [ 0.116015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.117010] ** ** [ 0.118008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.119011] ** ** [ 0.120010] ** This means that this kernel is built to expose internal ** [ 0.121010] ** IOMMU data structures, which may compromise security on ** [ 0.122011] ** your system. ** [ 0.123011] ** ** [ 0.124010] ** If you see this message and you are not debugging the ** [ 0.125018] ** kernel, report this immediately to your vendor! ** [ 0.126009] ** ** [ 0.127015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128012] ************************************************************* [ 0.129703] NET: Registered protocol family 16 [ 0.130437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.131055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.132064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.133672] cpuidle: using governor menu [ 0.134952] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137638] PCI: Using configuration type 1 for base access [ 0.139125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.150069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.151014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.154480] cryptd: max_cpu_qlen set to 1000 [ 0.157453] ACPI: Added _OSI(Module Device) [ 0.158000] ACPI: Added _OSI(Processor Device) [ 0.158000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.160011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.166669] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.178803] ACPI: Interpreter enabled [ 0.179074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.180015] ACPI: Using IOAPIC for interrupt routing [ 0.181097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.182383] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.198149] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.199037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.200019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.201076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.203421] acpiphp: Slot [2] registered [ 0.204169] acpiphp: Slot [5] registered [ 0.205165] acpiphp: Slot [6] registered [ 0.206189] acpiphp: Slot [7] registered [ 0.207131] acpiphp: Slot [8] registered [ 0.208160] acpiphp: Slot [9] registered [ 0.209121] acpiphp: Slot [10] registered [ 0.210160] acpiphp: Slot [3] registered [ 0.211117] acpiphp: Slot [4] registered [ 0.212093] acpiphp: Slot [11] registered [ 0.213100] acpiphp: Slot [12] registered [ 0.214146] acpiphp: Slot [13] registered [ 0.215105] acpiphp: Slot [14] registered [ 0.216221] acpiphp: Slot [15] registered [ 0.217150] acpiphp: Slot [16] registered [ 0.218097] acpiphp: Slot [17] registered [ 0.219096] acpiphp: Slot [18] registered [ 0.220093] acpiphp: Slot [19] registered [ 0.221314] acpiphp: Slot [20] registered [ 0.222111] acpiphp: Slot [21] registered [ 0.223124] acpiphp: Slot [22] registered [ 0.224123] acpiphp: Slot [23] registered [ 0.225098] acpiphp: Slot [24] registered [ 0.226154] acpiphp: Slot [25] registered [ 0.227197] acpiphp: Slot [26] registered [ 0.228092] acpiphp: Slot [27] registered [ 0.229129] acpiphp: Slot [28] registered [ 0.230090] acpiphp: Slot [29] registered [ 0.231074] acpiphp: Slot [30] registered [ 0.232086] acpiphp: Slot [31] registered [ 0.233074] PCI host bridge to bus 0000:00 [ 0.234023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.236021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.237023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.238041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.241026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.243207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.247259] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251676] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.264481] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.269051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.271018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.273017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.277018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.280987] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.285200] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.288101] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.293069] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.298014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.315524] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.322044] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.333790] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.344015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.353020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.376019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.389386] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.409017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.415021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.428016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.434000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.442014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.452015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.476016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.489332] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.500016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.509017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.544015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.559786] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.569016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.582014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.613018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.630120] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.642015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.654016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.690015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.707488] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.710450] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.712460] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.714492] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.716275] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.721547] iommu: Default domain type: Passthrough [ 0.722000] SCSI subsystem initialized [ 0.722000] ACPI: bus type USB registered [ 0.724119] usbcore: registered new interface driver usbfs [ 0.725229] usbcore: registered new interface driver hub [ 0.727078] usbcore: registered new device driver usb [ 0.729317] pps_core: LinuxPPS API ver. 1 registered [ 0.731016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735210] PTP clock support registered [ 0.737390] EDAC MC: Ver: 3.0.0 [ 0.738000] PCI: Using ACPI for IRQ routing [ 0.739199] NetLabel: Initializing [ 0.740007] NetLabel: domain hash size = 128 [ 0.742012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744074] NetLabel: unlabeled traffic allowed by default [ 0.746397] vgaarb: loaded [ 0.748501] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.750011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.760000] clocksource: Switched to clocksource kvm-clock [ 0.894620] VFS: Disk quotas dquot_6.6.0 [ 0.895940] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898836] *** VALIDATE ramfs *** [ 0.899821] *** VALIDATE hugetlbfs *** [ 0.901062] pnp: PnP ACPI init [ 0.903276] pnp: PnP ACPI: found 6 devices [ 0.919221] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.922294] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.924151] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.926081] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.928090] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.929966] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.932369] NET: Registered protocol family 2 [ 0.934492] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.939224] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.942247] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947389] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950329] TCP: Hash tables configured (established 65536 bind 65536) [ 0.952948] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.955369] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957812] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960195] NET: Registered protocol family 1 [ 0.962218] RPC: Registered named UNIX socket transport module. [ 0.964104] RPC: Registered udp transport module. [ 0.965430] RPC: Registered tcp transport module. [ 0.967067] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.969192] NET: Registered protocol family 44 [ 0.970480] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.972307] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.974147] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.976308] PCI: CLS 0 bytes, default 64 [ 0.978597] Unpacking initramfs... [ 2.717857] debug: unmapping init [mem 0xffff912bfcc54000-0xffff912bfffbffff] [ 2.722154] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.724645] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.727093] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.296831] Initialise system trusted keyrings [ 3.298480] Key type blacklist registered [ 3.300336] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.309983] zbud: loaded [ 3.313209] *** VALIDATE nfs *** [ 3.314482] *** VALIDATE nfs4 *** [ 3.316214] pstore: using deflate compression [ 3.320134] Platform Keyring initialized [ 3.435821] NET: Registered protocol family 38 [ 3.437770] Key type asymmetric registered [ 3.439492] Asymmetric key parser 'x509' registered [ 3.441819] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.444916] io scheduler mq-deadline registered [ 3.448715] io scheduler kyber registered [ 3.450481] io scheduler bfq registered [ 3.452332] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.455241] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.458428] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.461737] ACPI: Power Button [PWRF] [ 3.468198] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.556203] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.581390] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.600098] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.638254] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.666775] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.696212] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.702810] Non-volatile memory driver v1.3 [ 3.705178] Linux agpgart interface v0.103 [ 3.744406] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.747034] vda: detected capacity change from 0 to 68624384 [ 3.763843] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.766764] vdb: detected capacity change from 0 to 1073741824 [ 3.786100] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.789394] vdc: detected capacity change from 0 to 2621440000 [ 3.819358] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.825651] vdd: detected capacity change from 0 to 2621440000 [ 3.877291] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.880319] vde: detected capacity change from 0 to 4294967296 [ 3.915787] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.918698] vdf: detected capacity change from 0 to 4294967296 [ 3.946571] libphy: Fixed MDIO Bus: probed [ 3.959762] usbcore: registered new interface driver usbserial_generic [ 3.961520] usbserial: USB Serial support registered for generic [ 3.963310] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.966617] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.968080] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.970366] mousedev: PS/2 mouse device common for all mice [ 3.972912] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.975468] rtc_cmos 00:05: RTC can wake from S4 [ 3.979219] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.980927] rtc_cmos 00:05: registered as rtc0 [ 3.986018] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.988423] intel_pstate: CPU model not supported [ 3.991721] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.995386] hid: raw HID events driver (C) Jiri Kosina [ 3.997660] usbcore: registered new interface driver usbhid [ 3.999491] usbhid: USB HID core driver [ 4.000720] drop_monitor: Initializing network drop monitor service [ 4.002629] Initializing XFRM netlink socket [ 4.004159] NET: Registered protocol family 10 [ 4.006793] Segment Routing with IPv6 [ 4.007806] NET: Registered protocol family 17 [ 4.009723] mpls_gso: MPLS GSO support [ 4.013891] RAS: Correctable Errors collector initialized. [ 4.015791] AVX version of gcm_enc/dec engaged. [ 4.016963] AES CTR mode by8 optimization enabled [ 4.099181] sched_clock: Marking stable (4099150025, 0)->(5201751312, -1102601287) [ 4.103853] registered taskstats version 1 [ 4.106249] Loading compiled-in X.509 certificates [ 4.108930] zswap: loaded using pool lzo/zbud [ 4.136573] Key type big_key registered [ 4.149961] Key type encrypted registered [ 4.151700] ima: No TPM chip found, activating TPM-bypass! [ 4.154779] ima: Allocated hash algorithm: sha1 [ 4.157367] ima: No architecture policies found [ 4.159206] evm: Initialising EVM extended attributes: [ 4.161356] evm: security.selinux [ 4.162970] evm: security.ima [ 4.164782] evm: security.capability [ 4.166732] evm: HMAC attrs: 0x1 [ 4.170360] rtc_cmos 00:05: setting system clock to 2026-01-22 09:10:17 UTC (1769073017) [ 4.180592] debug: unmapping init [mem 0xffffffffaa603000-0xffffffffaa7fffff] [ 4.187781] debug: unmapping init [mem 0xffffffffa9382000-0xffffffffa9658fff] [ 4.200152] Write protecting the kernel read-only data: 28672k [ 4.205370] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 4.210381] debug: unmapping init [mem 0xffffffffa8314000-0xffffffffa83fffff] [ 4.265385] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.273962] systemd[1]: Detected virtualization kvm. [ 4.276801] systemd[1]: Detected architecture x86-64. [ 4.278894] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.313977] systemd[1]: No hostname configured. [ 4.316772] systemd[1]: Set hostname to . [ 4.320128] random: systemd: uninitialized urandom read (16 bytes read) [ 4.322781] systemd[1]: Initializing machine ID from random generator. [ 4.501633] random: ln: uninitialized urandom read (6 bytes read) [ 4.723181] random: systemd: uninitialized urandom read (16 bytes read) [ 4.725544] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.729873] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.735243] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.937988] device-mapper: uevent: version 1.0.3 [ 5.940387] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 8.032070] random: fast init done [ 8.169738] virtio_net virtio0 ens2: renamed from eth0 [ 8.242354] scsi host0: ata_piix [ 8.407473] scsi host1: ata_piix [ 8.409277] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.412042] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.833258] random: crng init done [ 14.836765] random: 7 urandom warning(s) missed due to ratelimiting [ 16.288553] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.937682] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.783303] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.597052] SELinux: Disabled at runtime. [ 21.678963] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.688702] systemd[1]: Detected virtualization kvm. [ 21.690983] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.113480] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.117667] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.129351] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.136817] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.142904] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.160933] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.173547] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ 23.706598] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.410693] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.645969] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.663362] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.631088] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.848609] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit)[ 33.954233] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 34.703358] NFS: Registering the id_resolver key type [ 34.706562] Key type id_resolver registered [ 34.708797] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 76.769757] libcfs: loading out-of-tree module taints kernel. [ 76.809718] Key type ._llcrypt registered [ 76.811242] Key type .llcrypt registered [ 76.915802] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 89.540782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 90.494220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.514117] alg: No test for adler32 (adler32-zlib) [ 91.723724] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 92.385539] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 94.039228] Key type lgssc registered [ 95.066469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.769386] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 106.770792] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.796472] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.564980] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.720658] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 132.636166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 141.219950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.268950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.281892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.476164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.497717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.544336] Lustre: lustre-MDT0000: new disk, initializing [ 142.602542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.613400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.510766] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.733488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.797030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 154.851380] Lustre: 6508:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.872628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.876291] Lustre: Skipped 1 previous similar message [ 154.938353] Lustre: lustre-MDT0001: new disk, initializing [ 154.970410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.987603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 155.006882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.926827] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.678995] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.626860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.670589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 167.831525] Lustre: lustre-OST0000: new disk, initializing [ 167.834448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.864581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.462664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.636997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 176.645372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 176.702534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.915532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.975739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 180.023312] Lustre: lustre-OST0001: new disk, initializing [ 180.027850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 180.059430] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.101694] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.349205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.361615] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 184.407874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 193.122932] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.043434] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 206.712914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 209.669291] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 212.263843] Lustre: DEBUG MARKER: Client: 2.17.50.4 [ 213.658347] Lustre: DEBUG MARKER: MDS: 2.17.50.4 [ 215.142021] Lustre: DEBUG MARKER: OSS: 2.17.50.4 [ 216.042310] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 22 04:13:47 EST 2026 [ 227.107386] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 228.002984] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 242.147026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 242.154339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 242.165536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 245.727995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.728708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 245.740561] Lustre: Skipped 3 previous similar messages [ 247.179715] LustreError: 12211:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 247.257481] Lustre: server umount lustre-MDT0000 complete [ 249.163722] LustreError: 6500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073262 with bad export cookie 1433195124994300226 [ 249.163996] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 249.168667] LustreError: 6500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 250.856302] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.856647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 250.874576] Lustre: Skipped 3 previous similar messages [ 250.881767] Lustre: Skipped 1 previous similar message [ 254.881803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 254.889943] Lustre: Skipped 1 previous similar message [ 255.317533] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 255.326838] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 255.399543] Lustre: server umount lustre-MDT0001 complete [ 257.848157] LustreError: 12613:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 257.851899] LustreError: 12613:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 257.884331] Lustre: server umount lustre-OST0000 complete [ 260.267684] LustreError: 12813:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 260.271044] LustreError: 12813:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 260.350873] Lustre: server umount lustre-OST0001 complete [ 264.005068] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 268.839773] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 273.842262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 277.636729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 280.793651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 284.192246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 289.456436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 289.565764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 289.846317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 289.887233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 289.946405] Lustre: lustre-MDT0000: new disk, initializing [ 290.044245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 290.069777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 295.324535] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 312.744169] hrtimer: interrupt took 3036659 ns [ 312.898928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.956618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 313.011575] Lustre: 14993:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 313.054748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 313.058215] Lustre: Skipped 1 previous similar message [ 313.107794] Lustre: lustre-MDT0001: new disk, initializing [ 313.180260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 313.215791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 313.228782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 318.262235] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 327.501967] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 331.971493] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 341.397211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.495144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 341.893593] Lustre: lustre-OST0000: new disk, initializing [ 341.904718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 342.052611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 343.063517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 343.084025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 343.372244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 350.137810] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 361.153410] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 366.629217] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 366.865703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 372.415594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 372.663973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 379.370681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 379.372467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.386399] LustreError: Skipped 1 previous similar message [ 379.389874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 379.433445] Lustre: Skipped 1 previous similar message [ 389.087240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 389.144132] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 389.148685] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 389.309342] Lustre: server umount lustre-OST0000 complete [ 394.725611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.742466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.745354] Lustre: Skipped 1 previous similar message [ 394.769316] Lustre: Skipped 4 previous similar messages [ 400.042836] LustreError: 18136:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 400.048962] LustreError: 18136:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 400.188339] Lustre: server umount lustre-MDT0000 complete [ 403.949389] LustreError: 14984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073417 with bad export cookie 1433195124994301745 [ 403.956454] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 403.961082] LustreError: 14984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 404.333729] Lustre: server umount lustre-MDT0001 complete [ 408.492654] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:16:59 (1769073419) [ 410.672874] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 412.419339] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:17:03 (1769073423) [ 418.479809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.980711] LustreError: 19009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 418.999228] LustreError: 19009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 419.052046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.418228] LustreError: 19008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 424.447549] LustreError: 19008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 424.500263] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 429.542354] LustreError: 19008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.509834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 438.155173] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 443.924485] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.623085] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 454.082630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 460.076994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 466.615153] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 475.108887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.120575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.136747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.152871] Lustre: Skipped 2 previous similar messages [ 479.200988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.634543] LustreError: 21253:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 480.646249] LustreError: 21253:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 480.940529] Lustre: server umount lustre-OST0000 complete [ 492.706351] Lustre: server umount lustre-MDT0000 complete [ 494.563225] LustreError: 20014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.596378] LustreError: 20014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 497.488294] LustreError: 18989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073510 with bad export cookie 1433195124994302627 [ 497.497363] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.501514] LustreError: 18989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 498.116949] Lustre: server umount lustre-MDT0001 complete [ 510.373595] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 514.250573] Key type lgssc unregistered [ 514.635500] LNet: 22192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.667468] LNetError: 22192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.700713] LNet: Removed LNI 192.168.203.155@tcp [ 516.020138] Key type .llcrypt unregistered [ 516.023176] Key type ._llcrypt unregistered [ 539.860831] Key type ._llcrypt registered [ 539.864979] Key type .llcrypt registered [ 539.990267] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 541.114978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 541.344037] alg: No test for adler32 (adler32-zlib) [ 542.468566] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 542.711576] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 544.399715] Key type lgssc registered [ 545.237489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.605181] loop: module loaded [ 552.674838] loop0: detected capacity change from 0 to 204800000 [ 552.943662] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 552.962781] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 553.260111] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 553.301307] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 558.129380] loop0: detected capacity change from 0 to 204800000 [ 558.341437] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 558.351848] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 558.362889] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 558.563069] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 563.117980] loop0: detected capacity change from 0 to 204800000 [ 568.332416] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 568.387358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 568.405413] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.727593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 569.802826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 569.904989] Lustre: lustre-MDT0000: new disk, initializing [ 570.007449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.034399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 574.049448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 582.893460] loop1: detected capacity change from 0 to 204800000 [ 592.010227] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 592.128423] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 592.629172] Lustre: lustre-OST07c6: new disk, initializing [ 592.633316] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 592.648486] Lustre: Skipped 1 previous similar message [ 592.795607] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 598.674540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 598.684128] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 598.717098] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 600.085864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 606.812501] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 606.818824] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 606.830771] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 606.839300] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 606.846058] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 606.854721] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 627.011958] Lustre: 24352:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 627.027172] Lustre: 24352:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 627.040594] Lustre: 24352:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 627.055753] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 627.070967] Lustre: 24352:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 627.088425] Lustre: 24352:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 632.514895] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 632.529900] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 632.547046] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 632.562761] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.574373] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 632.579313] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.585894] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 632.597711] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.608615] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 632.617288] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.629424] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 632.637077] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 637.445861] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 637.453249] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 637.457772] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 637.462457] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 637.468696] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 637.475087] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 637.481112] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 637.487367] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 637.501361] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 637.508695] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 637.513861] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 637.523519] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.370954] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 642.378910] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 642.390100] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 642.396178] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.408540] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 642.417346] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.427061] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 642.433161] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.439036] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 642.444338] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.451281] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 642.455876] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 653.554304] Lustre: 25998:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 653.570791] Lustre: 25998:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 653.579699] Lustre: 25998:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 653.591883] Lustre: 25998:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 653.602548] Lustre: 25998:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 653.614512] Lustre: 25998:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 653.622527] Lustre: 25998:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 653.632071] Lustre: 25998:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 653.639985] Lustre: 25998:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 653.646894] Lustre: 25998:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 653.652281] Lustre: 25998:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 653.665346] Lustre: 25998:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 701.417899] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 701.435382] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.460424] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 706.026639] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 707.260085] LustreError: 26537:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 707.439772] Lustre: server umount lustre-OST07c6 complete [ 717.324526] LustreError: 27039:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 717.340367] LustreError: 27039:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 717.557623] Lustre: server umount lustre-MDT0000 complete [ 733.247895] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 736.060818] Key type lgssc unregistered [ 736.418783] LNet: 27875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.430275] LNetError: 27875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.465251] LNet: Removed LNI 192.168.203.155@tcp [ 737.444698] Key type .llcrypt unregistered [ 737.451188] Key type ._llcrypt unregistered [ 754.658336] Key type ._llcrypt registered [ 754.663021] Key type .llcrypt registered [ 754.751306] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 755.690902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 755.801260] alg: No test for adler32 (adler32-zlib) [ 756.882203] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 757.073103] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 758.703188] Key type lgssc registered [ 760.107121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 768.603581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 768.639706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.392533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.717903] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 779.768360] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.794954] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 784.874640] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 785.578668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 786.153182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 790.574343] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 797.245461] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 802.213513] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 810.778861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 810.936295] LustreError: 30717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.429704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.547257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 817.638226] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 823.963904] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 841.190814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 841.198843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.219827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.222913] Lustre: Skipped 1 previous similar message [ 842.959313] LustreError: 31570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 843.120825] Lustre: server umount lustre-OST0000 complete [ 847.334052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 847.346971] LustreError: Skipped 1 previous similar message [ 847.351516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.361929] Lustre: Skipped 1 previous similar message [ 847.368899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.431093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.442601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.002960] LustreError: 31771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 853.013529] LustreError: 31771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 853.446240] Lustre: server umount lustre-MDT0000 complete [ 856.551518] LustreError: 29114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.575415] LustreError: 29114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 857.347317] LustreError: 29098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769073870 with bad export cookie 15342668705497034787 [ 857.357101] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 857.359771] LustreError: 29098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 857.567667] LustreError: 31973:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 857.574459] LustreError: 31973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 857.739774] Lustre: server umount lustre-MDT0001 complete [ 869.072893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 872.375300] Key type lgssc unregistered [ 872.678235] LNet: 32506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.682815] LNetError: 32506:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.693751] LNet: Removed LNI 192.168.203.155@tcp [ 873.361777] Key type .llcrypt unregistered [ 873.369394] Key type ._llcrypt unregistered [ 894.279980] Key type ._llcrypt registered [ 894.283392] Key type .llcrypt registered [ 894.432648] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 896.134754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.199303] alg: No test for adler32 (adler32-zlib) [ 897.365479] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 897.628830] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 899.327170] Key type lgssc registered [ 900.290245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 908.341496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 908.371288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.027957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.767911] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 920.547339] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.573924] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 923.504374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 924.057109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 929.393437] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 936.168765] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 941.166407] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 948.386541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.721704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.933385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 956.173798] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 962.193693] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 975.336630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 975.351388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.378253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.201235] LustreError: 36008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 979.512287] Lustre: server umount lustre-OST0000 complete [ 985.569171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.571149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.573299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 985.573307] LustreError: Skipped 1 previous similar message [ 985.582838] Lustre: Skipped 2 previous similar messages [ 985.594826] Lustre: Skipped 1 previous similar message [ 990.691613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.702474] Lustre: Skipped 2 previous similar messages [ 994.785248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.792303] Lustre: Skipped 1 previous similar message [ 998.369021] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 998.507604] LustreError: 36210:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 998.516742] LustreError: 36210:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 998.631842] Lustre: server umount lustre-MDT0000 complete [ 999.921612] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.950146] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1002.977065] LustreError: 36211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074016 with bad export cookie 1550931640884327559 [ 1002.980803] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.998214] LustreError: 36211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1003.163614] LustreError: 36414:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1003.176753] LustreError: 36414:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1003.382752] Lustre: server umount lustre-MDT0001 complete [ 1014.086225] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1017.051856] Key type lgssc unregistered [ 1017.398013] LNet: 36947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.410979] LNetError: 36947:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.431870] LNet: Removed LNI 192.168.203.155@tcp [ 1018.483250] Key type .llcrypt unregistered [ 1018.484638] Key type ._llcrypt unregistered [ 1046.450283] Key type ._llcrypt registered [ 1046.452132] Key type .llcrypt registered [ 1046.531365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1047.843078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.413174] alg: No test for adler32 (adler32-zlib) [ 1049.709630] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1050.113893] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1051.848977] Key type lgssc registered [ 1053.330043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.247276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.290473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.854201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.139669] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1074.147879] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.179117] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1078.867690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1079.266575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1079.453343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1083.710594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1089.806823] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1094.793251] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.890126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1103.289040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.519735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1109.652774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1115.537544] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1123.829496] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1124.849588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.871797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.874766] Lustre: Skipped 2 previous similar messages [ 1124.894760] Lustre: Skipped 2 previous similar messages [ 1128.925061] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1129.458919] LustreError: 40592:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1129.632889] Lustre: server umount lustre-MDT0000 complete [ 1129.953513] LustreError: 39460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.964489] LustreError: 39460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1134.046591] LustreError: 39460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.382845] LustreError: 38328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074147 with bad export cookie 2872909258931896236 [ 1134.394102] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.075700] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.109339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1139.104539] LustreError: 39460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.127974] LustreError: 39460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1140.627177] LustreError: 40795:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1140.630515] LustreError: 40795:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1140.888637] Lustre: server umount lustre-MDT0001 complete [ 1161.733078] LustreError: 41019:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1161.742490] LustreError: 41019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1161.969791] Lustre: server umount lustre-OST0000 complete [ 1180.237173] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1183.556955] Key type lgssc unregistered [ 1183.887660] LNet: 41858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.903259] LNetError: 41858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.935964] LNet: Removed LNI 192.168.203.155@tcp [ 1184.911406] Key type .llcrypt unregistered [ 1184.916205] Key type ._llcrypt unregistered [ 1207.993287] Key type ._llcrypt registered [ 1207.996100] Key type .llcrypt registered [ 1208.074694] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1208.674831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1208.701211] alg: No test for adler32 (adler32-zlib) [ 1209.833276] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1210.015439] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1211.671183] Key type lgssc registered [ 1212.784606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.681887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.706776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.240038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.482907] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1233.376396] LustreError: 43257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.399645] LustreError: 43257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1235.683928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1236.149187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1241.221625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1246.164360] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1250.472288] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1257.404958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1257.795670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.952274] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1268.028511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1269.294077] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1275.899900] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1277.413668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.430219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.447090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1278.436944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1278.443225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.456791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1278.463708] Lustre: Skipped 1 previous similar message [ 1280.569128] LustreError: 45503:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1280.694384] Lustre: server umount lustre-OST0000 complete [ 1303.009478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1303.013485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.034049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.387348] LustreError: 45879:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1305.398090] LustreError: 45879:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1305.489946] Lustre: server umount lustre-MDT0000 complete [ 1309.160954] LustreError: 43257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.190758] LustreError: 43257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1309.258112] LustreError: 43236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074322 with bad export cookie 5173164359050085599 [ 1309.267377] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1315.423216] LustreError: 46083:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1315.431671] LustreError: 46083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1315.634422] Lustre: server umount lustre-MDT0001 complete [ 1326.607888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1329.619409] Key type lgssc unregistered [ 1329.886987] LNet: 46617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.891103] LNetError: 46617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.909740] LNet: Removed LNI 192.168.203.155@tcp [ 1330.707795] Key type .llcrypt unregistered [ 1330.710135] Key type ._llcrypt unregistered [ 1353.965677] Key type ._llcrypt registered [ 1353.971397] Key type .llcrypt registered [ 1354.070260] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1355.146563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.217563] alg: No test for adler32 (adler32-zlib) [ 1356.332307] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1356.520726] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1358.183345] Key type lgssc registered [ 1359.097404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.599928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.612747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.242400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.642994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1377.763697] LustreError: 48016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.790902] LustreError: 48016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1381.752440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1382.282885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1387.215561] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1392.919521] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1397.513211] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1404.965976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1405.396601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.051633] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1413.638500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1416.161810] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1422.165957] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1425.693845] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1438.400584] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-22 4h34m08s [ 1440.358855] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1443.815262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1443.822866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.837861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.351348] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1444.366441] Lustre: Skipped 2 previous similar messages [ 1445.400584] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1445.406420] Lustre: Skipped 3 previous similar messages [ 1449.051898] LustreError: 50597:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1449.163781] Lustre: server umount lustre-MDT0000 complete [ 1450.459113] LustreError: 48010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.728465] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1456.600384] LustreError: 48011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.632265] LustreError: 48011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1459.485578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.591669] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.832563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.910478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1460.848416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1464.105667] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1467.557537] Lustre: DEBUG MARKER: Wait for df (36037) ... [ 1469.224449] Lustre: DEBUG MARKER: done [ 1475.042868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1475.051576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.072505] Lustre: Skipped 2 previous similar messages [ 1475.094909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.576646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.401203] LustreError: 51657:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1480.408799] LustreError: 51657:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1480.681648] Lustre: server umount lustre-OST0000 complete [ 1486.818195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.825156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1486.835219] Lustre: Skipped 1 previous similar message [ 1486.840128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1486.845550] Lustre: Skipped 3 previous similar messages [ 1490.668166] LustreError: 51858:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1490.677208] LustreError: 51858:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1490.937827] Lustre: server umount lustre-MDT0000 complete [ 1493.984710] LustreError: 48729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.008192] LustreError: 48729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1494.357512] LustreError: 50311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074507 with bad export cookie 12977727641210045911 [ 1494.368892] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.378879] LustreError: 50311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1494.581570] LustreError: 52059:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1494.589553] LustreError: 52059:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1494.976763] Lustre: server umount lustre-MDT0001 complete [ 1505.465460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1508.291864] Key type lgssc unregistered [ 1508.614698] LNet: 52591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.622059] LNetError: 52591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.645268] LNet: Removed LNI 192.168.203.155@tcp [ 1509.449986] Key type .llcrypt unregistered [ 1509.452307] Key type ._llcrypt unregistered [ 1540.862531] Key type ._llcrypt registered [ 1540.866327] Key type .llcrypt registered [ 1540.996361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1556.867267] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1558.210496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1558.275266] alg: No test for adler32 (adler32-zlib) [ 1559.334331] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1559.524232] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1561.239207] Key type lgssc registered [ 1562.397719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.419427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1576.579297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1582.749244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1589.016582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1596.736968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1596.832233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1596.842418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.145523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1598.190802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1598.298490] Lustre: lustre-MDT0000: new disk, initializing [ 1598.438793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.479228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1602.957760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1614.636959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1614.715960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1614.830329] Lustre: 56543:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1614.888687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1614.892299] Lustre: Skipped 1 previous similar message [ 1614.950913] Lustre: lustre-MDT0001: new disk, initializing [ 1615.038248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1615.076347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1615.108781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1619.000845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1626.336873] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1630.375447] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1637.703706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1637.766753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1638.047531] Lustre: lustre-OST0000: new disk, initializing [ 1638.060712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1638.197262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.658431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1644.666752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1644.671423] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1644.810366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1651.776621] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1656.497533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1656.702151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1661.304433] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1661.557617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1665.505650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1665.507065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.526892] LustreError: Skipped 1 previous similar message [ 1665.532619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1665.573314] Lustre: Skipped 1 previous similar message [ 1669.641348] LustreError: 59483:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1669.759791] Lustre: server umount lustre-OST0000 complete [ 1675.746865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.761197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.762067] Lustre: Skipped 1 previous similar message [ 1675.793150] Lustre: Skipped 2 previous similar messages [ 1680.098958] LustreError: 59685:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1680.103366] LustreError: 59685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1680.212526] Lustre: server umount lustre-MDT0000 complete [ 1683.882578] LustreError: 57323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074697 with bad export cookie 10672942397670461972 [ 1683.888688] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.898733] LustreError: 57323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1684.041936] LustreError: 59889:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1684.051704] LustreError: 59889:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1684.255579] Lustre: server umount lustre-MDT0001 complete [ 1692.408611] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 04:38:23 (1769074703) [ 1699.793091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1700.472285] LustreError: 60560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.506188] LustreError: 60560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1700.599817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.302266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1705.956275] LustreError: 60560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1710.058154] LustreError: 60560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.173975] LustreError: 60560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.246489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1715.745257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1720.627736] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1726.143071] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1729.676630] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1736.816865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1737.173121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.531775] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1747.530047] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1753.790771] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-22 4h39m24s [ 1755.419955] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1764.349590] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1767.361690] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1767.406552] LustreError: 60555:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1787.915457] Lustre: lustre-MDT0000: Client 99cdeafe-9e61-4642-8815-c0798298f168 (at 192.168.203.55@tcp) reconnecting [ 1807.455451] LustreError: 60555:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1809.686744] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1811.814202] Lustre: DEBUG MARKER: done [ 1817.064589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1817.065863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.102253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.126562] Lustre: Skipped 1 previous similar message [ 1817.157037] Lustre: Skipped 1 previous similar message [ 1822.184460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.202221] Lustre: Skipped 1 previous similar message [ 1822.841332] LustreError: 63565:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1822.853702] LustreError: 63565:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1822.958387] Lustre: server umount lustre-OST0000 complete [ 1827.298382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.303649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.316618] Lustre: Skipped 1 previous similar message [ 1832.449474] LustreError: 63768:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1832.457949] LustreError: 63768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1832.638074] Lustre: server umount lustre-MDT0000 complete [ 1833.441234] LustreError: 60555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.457547] LustreError: 60555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1836.441650] LustreError: 60540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769074849 with bad export cookie 10672942397670462854 [ 1836.444984] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1836.449249] LustreError: 60540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1836.810666] Lustre: server umount lustre-MDT0001 complete [ 1847.192528] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1850.088244] Key type lgssc unregistered [ 1850.343499] LNet: 64501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.350654] LNetError: 64501:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.367557] LNet: Removed LNI 192.168.203.155@tcp [ 1852.216091] Key type .llcrypt unregistered [ 1852.217962] Key type ._llcrypt unregistered [ 1883.487963] Key type ._llcrypt registered [ 1883.492170] Key type .llcrypt registered [ 1883.615508] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1897.371585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1898.426533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1898.575624] alg: No test for adler32 (adler32-zlib) [ 1899.676852] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 1899.897785] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1901.583444] Key type lgssc registered [ 1902.673925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1910.411186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1916.595035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1922.682572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.818386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.092716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.148985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1935.159747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.326299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1936.351744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1936.439726] Lustre: lustre-MDT0000: new disk, initializing [ 1936.578958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.609404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1940.897532] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1950.994975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1951.071258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1951.173416] Lustre: 68453:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1951.260064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1951.265532] Lustre: Skipped 1 previous similar message [ 1951.327970] Lustre: lustre-MDT0001: new disk, initializing [ 1951.427136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1951.481370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1951.488296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1955.303618] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1961.816583] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1965.460202] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.718690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1972.815638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1973.119492] Lustre: lustre-OST0000: new disk, initializing [ 1973.127749] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1973.199357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.589131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1980.604525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1980.729785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1982.042288] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1990.979938] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1996.485864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1996.769365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2001.009559] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2001.303568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2006.496515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.506742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2006.509270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2006.518966] Lustre: Skipped 1 previous similar message [ 2006.522679] LustreError: Skipped 1 previous similar message [ 2009.344689] LustreError: 71392:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2009.615817] Lustre: server umount lustre-OST0000 complete [ 2016.737875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.740761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.755297] Lustre: Skipped 1 previous similar message [ 2016.763633] Lustre: Skipped 2 previous similar messages [ 2020.070517] LustreError: 71595:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2020.085381] LustreError: 71595:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2020.252729] Lustre: server umount lustre-MDT0000 complete [ 2021.858318] LustreError: 68463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.874166] LustreError: 68463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2025.893145] LustreError: 69232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075039 with bad export cookie 18354467826797185015 [ 2025.898640] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.914092] LustreError: 69232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2026.186135] LustreError: 71797:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2026.193879] LustreError: 71797:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2026.572698] Lustre: server umount lustre-MDT0001 complete [ 2036.627425] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 04:44:07 (1769075047) [ 2038.664330] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 2040.938915] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 04:44:11 (1769075051) [ 2051.460689] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2054.864867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.134801] LustreError: 72847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.174400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.990761] Lustre: Failing over lustre-MDT0000 [ 2055.996599] LustreError: 72875:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2056.002937] LustreError: 72875:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2056.007984] LustreError: 72875:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 2056.095895] LustreError: 72926:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2056.100365] LustreError: 72926:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2056.237404] Lustre: server umount lustre-MDT0000 complete [ 2061.681703] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:44:32 (1769075072) [ 2067.821947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.449300] LustreError: 73496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.517862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.071902] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2073.570171] LustreError: 73496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.691587] LustreError: 73496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.091414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2082.728032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.380906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2094.674130] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.888488] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2105.409568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2105.826266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.723807] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2117.690634] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2122.950533] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2133.984098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2133.990157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.015490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.816135] LustreError: 75838:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2135.823111] LustreError: 75838:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2135.988463] Lustre: server umount lustre-MDT0000 complete [ 2139.625756] LustreError: 73490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.643453] LustreError: 73490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2139.668098] LustreError: 73475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075152 with bad export cookie 18354467826797186170 [ 2139.685305] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.689155] LustreError: 73475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2139.948536] Lustre: server umount lustre-MDT0001 complete [ 2143.129851] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2147.653265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2155.586286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.119701] Lustre: 66666:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075158/real 1769075158] req@0000000052a656b7 x1855009496051712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769075174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2161.162516] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.180268] Lustre: Skipped 2 previous similar messages [ 2165.284804] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfeb82b3e099513ea [ 2165.295607] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 2165.603068] LustreError: 75102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.655385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.720255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2166.687197] Lustre: 66665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075163/real 1769075163] req@00000000d145047e x1855009496051840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769075179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2169.811104] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2170.787500] Lustre: 66664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075168/real 1769075168] req@00000000765bd450 x1855009496052096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769075184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.841346] Lustre: 66664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075173/real 1769075173] req@0000000040f3cbf5 x1855009496052224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769075189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.918297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2182.996625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2183.659735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2188.686254] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2189.798055] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2192.314913] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2199.009484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2199.022996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.045811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.048819] Lustre: Skipped 3 previous similar messages [ 2200.041979] LustreError: 78304:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2200.048717] LustreError: 78304:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2200.071362] LustreError: 75102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.090567] LustreError: 75102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2200.243674] Lustre: server umount lustre-MDT0000 complete [ 2203.848572] LustreError: 73477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075217 with bad export cookie 18354467826797188074 [ 2203.862924] LustreError: 73477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2203.866290] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.231859] Lustre: server umount lustre-MDT0001 complete [ 2211.352767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2220.707556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.537131] Lustre: 66665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075218/real 1769075218] req@0000000097979d57 x1855009496077056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769075234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2221.570733] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.587624] Lustre: Skipped 2 previous similar messages [ 2229.215871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2230.176122] Lustre: 66665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769075227/real 1769075227] req@00000000a0f93f02 x1855009496077568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769075243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2230.201145] Lustre: 66665:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2233.644968] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2240.487059] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2242.891804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2243.451120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2243.458275] Lustre: Skipped 2 previous similar messages [ 2248.041405] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2248.684206] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2253.482212] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2257.604402] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2269.152538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2269.153473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.158950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.158959] Lustre: Skipped 2 previous similar messages [ 2269.196504] Lustre: Skipped 2 previous similar messages [ 2270.709635] LustreError: 80762:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2270.718885] LustreError: 80762:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2270.833720] Lustre: server umount lustre-MDT0000 complete [ 2274.239635] LustreError: 73477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075287 with bad export cookie 18354467826797189446 [ 2274.241880] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.265715] LustreError: 73477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2274.311048] LustreError: 79748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.334277] LustreError: 79748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2274.698431] Lustre: server umount lustre-MDT0001 complete [ 2280.735764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2284.244099] Lustre: DEBUG MARKER: files compared the same [ 2286.863254] Lustre: server umount lustre-OST0000 complete [ 2302.620723] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2305.553920] Key type lgssc unregistered [ 2305.874891] LNet: 82395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.882669] LNetError: 82395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.920741] LNet: Removed LNI 192.168.203.155@tcp [ 2306.849225] Key type .llcrypt unregistered [ 2306.851834] Key type ._llcrypt unregistered [ 2332.186741] Key type ._llcrypt registered [ 2332.193289] Key type .llcrypt registered [ 2332.357561] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2333.396649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2333.440399] alg: No test for adler32 (adler32-zlib) [ 2334.871437] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2335.170728] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2336.879123] Key type lgssc registered [ 2338.102686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2344.207592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2344.232691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.732940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.823825] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2355.169812] LustreError: 83796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.180501] LustreError: 83796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2358.122477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2358.383788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2361.912553] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2367.179461] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2370.953424] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2378.084966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2378.386052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.952273] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2390.731040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2392.190591] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2404.323930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.335314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.355920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.369457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.370021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.245681] LustreError: 86044:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2408.572523] Lustre: server umount lustre-OST0000 complete [ 2414.560311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.578277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.591451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.598625] Lustre: Skipped 1 previous similar message [ 2416.619320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.620882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.729917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.738288] Lustre: Skipped 2 previous similar messages [ 2427.359226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2427.443457] LustreError: 86246:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2427.453692] LustreError: 86246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2427.524921] Lustre: server umount lustre-MDT0000 complete [ 2430.944568] LustreError: 83793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.979539] LustreError: 83793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2431.265532] LustreError: 83778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075444 with bad export cookie 6293440555429361236 [ 2431.272591] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.278767] LustreError: 83778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2431.484441] LustreError: 86448:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2431.500918] LustreError: 86448:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2431.778101] Lustre: server umount lustre-MDT0001 complete [ 2444.170300] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2447.867887] Key type lgssc unregistered [ 2448.284723] LNet: 86984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2448.303042] LNetError: 86984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.383092] LNet: Removed LNI 192.168.203.155@tcp [ 2450.739983] Key type .llcrypt unregistered [ 2450.741439] Key type ._llcrypt unregistered [ 2475.503890] Key type ._llcrypt registered [ 2475.506343] Key type .llcrypt registered [ 2475.681471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2477.068226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.121098] alg: No test for adler32 (adler32-zlib) [ 2478.106815] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2478.273342] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2479.935206] Key type lgssc registered [ 2481.346732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2489.019600] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2489.057948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2504.736264] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2509.855243] LustreError: 88354:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2536.607603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2542.313977] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2547.570662] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2554.425296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.506341] Lustre: *** cfs_fail_loc=706, val=0*** [ 2554.798033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.876201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2559.262870] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2559.984340] LustreError: 88380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.002880] LustreError: 88380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2565.095706] LustreError: 88380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.512557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2568.975555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.417961] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2578.721995] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2582.199332] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2584.548327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2584.561674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.587781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2585.571324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2585.579130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.594599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2586.612215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.467138] LustreError: 90681:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2590.566700] Lustre: server umount lustre-OST0000 complete [ 2594.784958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2594.794361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.806374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.816818] Lustre: Skipped 1 previous similar message [ 2596.843958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.954292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.958095] Lustre: Skipped 3 previous similar messages [ 2608.095214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2608.225943] LustreError: 90882:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2608.238960] LustreError: 90882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2608.313106] Lustre: server umount lustre-MDT0000 complete [ 2611.687864] LustreError: 88361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769075625 with bad export cookie 9328926654650287147 [ 2611.712940] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.831752] LustreError: 91084:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2611.837771] LustreError: 91084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2611.971723] Lustre: server umount lustre-MDT0001 complete [ 2622.331551] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2625.633484] Key type lgssc unregistered [ 2626.122366] LNet: 91617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.134171] LNetError: 91617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.158228] LNet: Removed LNI 192.168.203.155@tcp [ 2627.326801] Key type .llcrypt unregistered [ 2627.340306] Key type ._llcrypt unregistered [ 2652.232577] Key type ._llcrypt registered [ 2652.235127] Key type .llcrypt registered [ 2652.329597] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2653.228031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2653.288181] alg: No test for adler32 (adler32-zlib) [ 2654.374440] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 2654.538124] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2656.183453] Key type lgssc registered [ 2657.519476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2665.817751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2665.841625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.996495] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2680.039924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2680.824944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2685.332248] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2685.921509] LustreError: 93686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.931636] LustreError: 93686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2691.046371] LustreError: 93690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.056342] LustreError: 93690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2692.734525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2692.961847] LustreError: 93687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.985320] LustreError: 93687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2693.014867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.211145] LustreError: 93691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.227561] LustreError: 93691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2698.242584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2703.333921] LustreError: 94418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.357265] LustreError: 94418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2705.805517] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.823691] Lustre: MGS: Client fc5aaa7d-d190-4a42-ba16-31af5b057b68 (at 0@lo) reconnecting [ 2706.044603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.107207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2710.718875] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2726.369508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2726.386057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.415196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.909333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.925306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2727.937793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.943424] LustreError: 95661:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2731.030066] Lustre: server umount lustre-OST0000 complete [ 2738.145767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.148327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.162696] Lustre: Skipped 1 previous similar message [ 2738.194583] Lustre: Skipped 2 previous similar messages [ 2740.378113] LustreError: 95863:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2740.390339] LustreError: 95863:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2740.480161] Lustre: server umount lustre-MDT0000 complete [ 2743.273905] LustreError: 94418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.304163] LustreError: 94418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2744.506487] LustreError: 96066:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2744.511369] LustreError: 96066:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2744.767639] Lustre: server umount lustre-MDT0001 complete [ 2748.284305] LustreError: 96266:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2748.288700] LustreError: 96266:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2748.363637] Lustre: server umount MGS complete [ 2760.052842] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:56:11 (1769075771) [ 2790.221698] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2800.666181] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2810.554379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2816.878424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2823.173256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2830.427532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2837.385928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2837.458383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.990481] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2851.613714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.674132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2851.847493] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2851.892642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2851.970482] Lustre: lustre-MDT0001: new disk, initializing [ 2852.048644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.076198] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2852.086860] Lustre: 100785:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2852.095982] LustreError: 100785:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2856.428874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2867.232953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2867.298789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2867.538257] Lustre: lustre-OST0000: new disk, initializing [ 2867.549822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2867.652914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.295933] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2876.001200] Lustre: 101948:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004eafbe16 x1855010286641280/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769075905 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2882.674943] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2890.051255] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2890.068913] Lustre: MGS: Client a33504bd-877b-4302-95af-ca9ad9fa4db9 (at 0@lo) reconnecting [ 2890.128851] Lustre: 100070:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2890.161935] LustreError: 100070:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2890.198285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2890.263756] Lustre: lustre-MDT0000: new disk, initializing [ 2890.454140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.489888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2890.527809] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2891.558722] Lustre: 102499:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000549042a4 x1855010286647552/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1769075920 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2896.264323] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2896.410529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2896.433376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2896.700666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2922.464031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2922.468539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.471460] LustreError: Skipped 1 previous similar message [ 2922.472524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2922.513400] Lustre: Skipped 1 previous similar message [ 2927.077918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2927.085956] Lustre: Skipped 2 previous similar messages [ 2936.288032] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2936.326278] LustreError: 103021:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2936.336258] LustreError: 103021:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2936.428161] Lustre: server umount lustre-OST0000 complete [ 2942.433953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2942.434154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.443701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.443708] Lustre: Skipped 2 previous similar messages [ 2942.474607] Lustre: Skipped 1 previous similar message [ 2945.728929] LustreError: 103223:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2945.740161] LustreError: 103223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2945.787042] Lustre: server umount lustre-MDT0000 complete [ 2947.553106] LustreError: 100756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.572122] LustreError: 100756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2956.107908] Lustre: server umount lustre-MDT0001 complete [ 2960.294911] Lustre: server umount MGS complete [ 2971.444906] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:59:42 (1769075982) [ 2992.321393] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2995.555329] Key type lgssc unregistered [ 2995.871376] LNet: 104965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2995.879819] LNetError: 104965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2995.908156] LNet: Removed LNI 192.168.203.155@tcp [ 2996.850435] Key type .llcrypt unregistered [ 2996.857348] Key type ._llcrypt unregistered [ 3016.042576] Key type ._llcrypt registered [ 3016.044907] Key type .llcrypt registered [ 3016.215931] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3017.748026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3017.943043] alg: No test for adler32 (adler32-zlib) [ 3019.022353] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3019.260327] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3020.959182] Key type lgssc registered [ 3022.577606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3031.452844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3031.479243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.130602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.421756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3043.300266] LustreError: 106188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.335096] LustreError: 106188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3047.580599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3047.974683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3053.498891] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3060.481625] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3064.547862] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3072.719723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3073.225516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.396283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3080.519421] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3086.720418] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3099.107222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.120320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3099.137191] Lustre: Skipped 1 previous similar message [ 3099.165592] Lustre: Skipped 1 previous similar message [ 3101.593368] LustreError: 108449:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3101.998512] Lustre: server umount lustre-OST0000 complete [ 3109.346070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3109.352525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.369476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3109.377990] Lustre: Skipped 1 previous similar message [ 3109.405458] Lustre: Skipped 1 previous similar message [ 3112.100950] LustreError: 108652:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3112.109886] LustreError: 108652:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3112.218949] Lustre: server umount lustre-MDT0000 complete [ 3114.466709] LustreError: 108375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.486302] LustreError: 108375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3116.098340] LustreError: 106168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076129 with bad export cookie 11867200760620129746 [ 3116.101279] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3116.114195] LustreError: 106168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3116.302572] LustreError: 108855:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3116.323539] LustreError: 108855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3116.619071] Lustre: server umount lustre-MDT0001 complete [ 3127.439889] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3130.283773] Key type lgssc unregistered [ 3130.558329] LNet: 109386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3130.583173] LNetError: 109386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3131.626358] LNet: Removed LNI 192.168.203.155@tcp [ 3132.477902] Key type .llcrypt unregistered [ 3132.480338] Key type ._llcrypt unregistered [ 3148.387646] Key type ._llcrypt registered [ 3148.392470] Key type .llcrypt registered [ 3148.494614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3149.643343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.753685] alg: No test for adler32 (adler32-zlib) [ 3150.803737] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3151.005205] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3152.663162] Key type lgssc registered [ 3153.644487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3159.448353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3159.453721] LustreError: 110451:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 3159.458892] LustreError: 110452:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 3159.463872] LustreError: 110451:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3159.469670] LustreError: 110451:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3159.476747] LustreError: 110451:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3159.481350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.085495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.800498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3169.117596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3174.022238] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3181.751516] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lsmod [ 3184.358322] LustreError: 111607:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 3184.614040] LustreError: 111630:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 3184.629498] LustreError: 111607:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4737 [ 3184.637808] LustreError: 111607:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3184.646215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3184.649056] LustreError: 111607:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3185.173054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.239331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3189.728546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3189.745227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.760786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.266126] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.266657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.289193] Lustre: Skipped 1 previous similar message [ 3191.316733] Lustre: Skipped 2 previous similar messages [ 3194.406859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.563302] LustreError: 111817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3194.699547] Lustre: server umount lustre-MDT0000 complete [ 3198.791400] LustreError: 110468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076212 with bad export cookie 466456543345781266 [ 3198.816168] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3198.816972] LustreError: 110468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3199.467872] LustreError: 111669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.469648] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.501239] LustreError: 111669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3199.537431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3199.546783] Lustre: Skipped 2 previous similar messages [ 3200.671281] LustreError: 110924:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001ca3c87d x1855010807765376/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3200.718750] LustreError: 112021:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3200.741654] LustreError: 112021:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3200.930804] Lustre: server umount lustre-MDT0001 complete [ 3205.131422] LustreError: 112223:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3205.136460] LustreError: 112223:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3205.249691] Lustre: server umount lustre-OST0000 complete [ 3213.239445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.767515] LustreError: 112699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.895969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.898131] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3218.915317] LustreError: 112699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.038225] LustreError: 112699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.587464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3226.830175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3231.685430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3237.027633] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3240.297938] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3248.143859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3248.597533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.960813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3258.872636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3262.678402] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3272.160970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3272.177557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.203840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3274.220490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.323938] LustreError: 114945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3278.336638] LustreError: 114945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3278.789666] Lustre: server umount lustre-OST0000 complete [ 3283.425142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3283.434797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.460650] Lustre: Skipped 1 previous similar message [ 3283.466109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.468937] Lustre: Skipped 2 previous similar messages [ 3297.247262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3297.354086] LustreError: 115147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3297.358755] LustreError: 115147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3297.532760] Lustre: server umount lustre-MDT0000 complete [ 3298.802086] LustreError: 112693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.822696] LustreError: 112693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3301.906196] LustreError: 112679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076315 with bad export cookie 466456543345782197 [ 3301.923585] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.924886] LustreError: 112679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3302.355738] Lustre: server umount lustre-MDT0001 complete [ 3313.762946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3316.824595] Key type lgssc unregistered [ 3317.223640] LNet: 115883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3317.229197] LNetError: 115883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3317.246197] LNet: Removed LNI 192.168.203.155@tcp [ 3318.236389] Key type .llcrypt unregistered [ 3318.237939] Key type ._llcrypt unregistered [ 3346.070834] Key type ._llcrypt registered [ 3346.074644] Key type .llcrypt registered [ 3346.205792] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3347.476194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3347.504175] alg: No test for adler32 (adler32-zlib) [ 3348.765469] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3348.980858] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3350.631359] Key type lgssc registered [ 3352.331142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.259414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.279884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.850881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.147809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3372.008046] LustreError: 117279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.054069] LustreError: 117279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3376.143451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3376.556160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.596019] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3386.632328] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3391.221790] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3398.955932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.524106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.783714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3407.176455] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3414.995541] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3422.018054] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3427.005193] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3437.550500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3437.553847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.562120] LustreError: Skipped 1 previous similar message [ 3437.574077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3437.576017] Lustre: Skipped 1 previous similar message [ 3437.603313] Lustre: Skipped 1 previous similar message [ 3439.170910] LustreError: 119634:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3439.448397] Lustre: server umount lustre-OST0000 complete [ 3447.776468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.795105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.987608] LustreError: 119835:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3450.003737] LustreError: 119835:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3450.279298] Lustre: server umount lustre-MDT0000 complete [ 3452.897884] LustreError: 117274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.909646] LustreError: 117274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3454.548181] LustreError: 117261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076467 with bad export cookie 2000594904254930116 [ 3454.552940] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.564846] LustreError: 117261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3454.735185] LustreError: 120038:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3454.749734] LustreError: 120038:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3455.189226] Lustre: server umount lustre-MDT0001 complete [ 3469.840615] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3473.556579] Key type lgssc unregistered [ 3473.800652] LNet: 120572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3473.809502] LNetError: 120572:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3474.857206] LNet: Removed LNI 192.168.203.155@tcp [ 3475.710781] Key type .llcrypt unregistered [ 3475.713694] Key type ._llcrypt unregistered [ 3493.618835] Key type ._llcrypt registered [ 3493.621082] Key type .llcrypt registered [ 3493.842345] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3494.853496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3494.985294] alg: No test for adler32 (adler32-zlib) [ 3496.275664] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3496.632935] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3498.407197] Key type lgssc registered [ 3500.126580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.562480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3509.591899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.093062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.834420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3521.508430] LustreError: 121776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.556447] LustreError: 121776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3526.631792] LustreError: 121776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.981986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3527.715477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3533.722330] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3541.401491] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3546.139520] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3554.485151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3554.995514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.039164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3562.003690] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3568.078766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3582.944079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3582.947434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.970874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3583.013930] Lustre: Skipped 1 previous similar message [ 3588.066295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.074634] Lustre: Skipped 1 previous similar message [ 3588.309021] LustreError: 124048:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3588.532912] Lustre: server umount lustre-OST0000 complete [ 3593.185467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.192646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.195487] Lustre: Skipped 1 previous similar message [ 3598.305678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.318425] Lustre: Skipped 1 previous similar message [ 3599.211498] LustreError: 124249:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3599.219064] LustreError: 124249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3599.338774] Lustre: server umount lustre-MDT0000 complete [ 3603.018033] LustreError: 121757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076616 with bad export cookie 10642891749032323051 [ 3603.031209] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3603.173809] LustreError: 124451:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3603.189912] LustreError: 124451:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3603.332812] Lustre: server umount lustre-MDT0001 complete [ 3615.798216] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3618.203777] Key type lgssc unregistered [ 3618.493885] LNet: 124984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3618.506592] LNetError: 124984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3618.535368] LNet: Removed LNI 192.168.203.155@tcp [ 3619.503188] Key type .llcrypt unregistered [ 3619.504877] Key type ._llcrypt unregistered [ 3650.492694] Key type ._llcrypt registered [ 3650.495988] Key type .llcrypt registered [ 3650.642785] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3652.127312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3652.208141] alg: No test for adler32 (adler32-zlib) [ 3653.812637] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3654.150852] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3655.920445] Key type lgssc registered [ 3657.620186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3667.773101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3667.808108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.609495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.673665] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3678.688664] LustreError: 126384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3678.700421] LustreError: 126384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3683.817364] LustreError: 126384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3684.706680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3685.376386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3690.546259] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3696.489095] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3700.860703] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3709.175836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3709.471602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.630472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3716.990370] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3723.814477] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3730.853032] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3734.779089] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3734.784320] Lustre: Skipped 2 previous similar messages [ 3736.687889] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3736.721445] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3738.745766] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3738.751651] Lustre: Skipped 2 previous similar messages [ 3744.391798] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3744.397646] Lustre: Skipped 1 previous similar message [ 3751.162068] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3751.180017] Lustre: Skipped 2 previous similar messages [ 3751.992624] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp [ 3764.602767] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.55/24@tcp in log lustre-MDT0000 [ 3764.610889] Lustre: Skipped 2 previous similar messages [ 3771.594220] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3771.599127] Lustre: Skipped 1 previous similar message [ 3778.054511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3778.054829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3778.073135] LustreError: Skipped 3 previous similar messages [ 3778.134974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3779.346146] LustreError: 129950:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3779.618469] Lustre: server umount lustre-OST0000 complete [ 3786.208717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.213209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.241796] Lustre: Skipped 1 previous similar message [ 3786.262175] Lustre: Skipped 1 previous similar message [ 3790.319311] LustreError: 130151:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3790.331437] LustreError: 130151:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3790.536218] Lustre: server umount lustre-MDT0000 complete [ 3791.333804] LustreError: 126381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.359126] LustreError: 126381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3794.448197] LustreError: 126366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076807 with bad export cookie 17664203633619495421 [ 3794.449913] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.459335] LustreError: 126366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3804.653782] LustreError: 130352:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3804.671360] LustreError: 130352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3804.908536] Lustre: server umount lustre-MDT0001 complete [ 3818.755412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3822.397340] Key type lgssc unregistered [ 3822.793902] LNet: 130886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3822.812583] LNetError: 130886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3822.841181] LNet: Removed LNI 192.168.203.155@tcp [ 3824.205925] Key type .llcrypt unregistered [ 3824.208958] Key type ._llcrypt unregistered [ 3858.028640] Key type ._llcrypt registered [ 3858.031824] Key type .llcrypt registered [ 3858.225862] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3859.644565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3859.934142] alg: No test for adler32 (adler32-zlib) [ 3860.906707] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 3861.118523] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3862.775315] Key type lgssc registered [ 3863.995616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3872.180738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3872.209996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.811319] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3873.842740] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.55/24@tcp [ 3873.870470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.336125] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3884.005495] LustreError: 132432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.035595] LustreError: 132432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3888.767037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3889.142104] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3889.286491] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3889.316096] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3889.383950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3894.528877] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3900.672806] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3905.268880] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3913.740467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3914.129935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.232884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3920.596410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3926.647498] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3939.827684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.827863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3939.850209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.876967] Lustre: Skipped 1 previous similar message [ 3943.561518] LustreError: 134736:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3944.100852] Lustre: server umount lustre-OST0000 complete [ 3951.075786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3951.100302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.119876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3951.142403] Lustre: Skipped 1 previous similar message [ 3955.694604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3955.718926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.530956] LustreError: 134938:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3956.543038] LustreError: 134938:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3956.764906] Lustre: server umount lustre-MDT0000 complete [ 3959.913853] LustreError: 134058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.986562] LustreError: 134058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3961.770431] LustreError: 132413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769076975 with bad export cookie 17958974394506129014 [ 3961.777125] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.799497] LustreError: 132413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3962.071624] LustreError: 135140:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3962.088835] LustreError: 135140:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3962.628918] Lustre: server umount lustre-MDT0001 complete [ 3976.268379] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3980.734990] Key type lgssc unregistered [ 3981.231830] LNet: 135674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3981.250356] LNetError: 135674:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3981.275766] LNet: Removed LNI 192.168.203.155@tcp [ 3982.408508] Key type .llcrypt unregistered [ 3982.412974] Key type ._llcrypt unregistered [ 4010.887577] Key type ._llcrypt registered [ 4010.891454] Key type .llcrypt registered [ 4011.030879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4026.385544] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4028.000774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4028.018093] alg: No test for adler32 (adler32-zlib) [ 4029.229993] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 4029.447395] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4031.167531] Key type lgssc registered [ 4033.029510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4042.942261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4049.287916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.317622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4063.844941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.838285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.981834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4074.011386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.350332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4075.386336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4075.459238] Lustre: lustre-MDT0000: new disk, initializing [ 4075.602355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4075.637337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4081.691640] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4094.896789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.964490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4095.111985] Lustre: 139053:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4095.184424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4095.192630] Lustre: Skipped 1 previous similar message [ 4095.297141] Lustre: lustre-MDT0001: new disk, initializing [ 4095.429446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4095.472726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4095.490796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4100.973300] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4110.232615] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4114.610779] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4123.286330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4123.343460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4123.724331] Lustre: lustre-OST0000: new disk, initializing [ 4123.734174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4123.873176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.465959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4130.476926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4130.651625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4131.196165] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4141.707049] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4147.805954] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4148.140398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4153.158204] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4153.434766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4156.384351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4156.386100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4156.388300] LustreError: Skipped 1 previous similar message [ 4156.397833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4156.418647] Lustre: Skipped 1 previous similar message [ 4156.448596] Lustre: Skipped 1 previous similar message [ 4161.411512] LustreError: 141994:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4161.849675] Lustre: server umount lustre-OST0000 complete [ 4166.624247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.645159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4166.654466] Lustre: Skipped 1 previous similar message [ 4171.745884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.237622] LustreError: 142197:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4172.242141] LustreError: 142197:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4172.402449] Lustre: server umount lustre-MDT0000 complete [ 4176.723233] LustreError: 140809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077190 with bad export cookie 13320509036146866604 [ 4176.729688] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4176.739563] LustreError: 140809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4176.924632] LustreError: 142399:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4176.941847] LustreError: 142399:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4177.302922] Lustre: server umount lustre-MDT0001 complete [ 4188.055070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.830675] LustreError: 142875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4188.866440] LustreError: 142875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4188.969684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.273794] LustreError: 142875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.910565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4199.393383] LustreError: 142875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.511939] LustreError: 142875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.238822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4208.730463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4214.486654] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4221.353331] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4225.616299] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4233.415867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4233.902043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.443579] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4246.481921] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4251.726167] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4252.091989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4257.460287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4257.669872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4268.004300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4268.132736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4268.298915] Lustre: lustre-OST0001: new disk, initializing [ 4268.306626] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4268.456426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4275.294969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4275.299265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4275.438290] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4275.548171] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4284.842797] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4285.140990] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4291.004365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4291.247806] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4295.182526] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa02087539800.ost_server_uuid 50 [ 4296.770285] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa02087539800.ost_server_uuid in IDLE state after 0 sec [ 4306.408370] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4306.418941] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.448462] Lustre: Skipped 1 previous similar message [ 4306.472216] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4306.479520] Lustre: Skipped 1 previous similar message [ 4311.231536] LustreError: 147127:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4311.240718] LustreError: 147127:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4311.408865] Lustre: server umount lustre-OST0001 complete [ 4311.531759] LustreError: 146203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.557161] LustreError: 146203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4316.641098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.650468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4316.651407] Lustre: Skipped 2 previous similar messages [ 4316.671857] Lustre: Skipped 1 previous similar message [ 4321.467881] LustreError: 147329:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4321.482645] LustreError: 147329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4321.709730] Lustre: server umount lustre-OST0000 complete [ 4326.379663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4326.394538] LustreError: Skipped 1 previous similar message [ 4326.404562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.428213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.444638] LustreError: 147531:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4332.461383] LustreError: 147531:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4332.791192] Lustre: server umount lustre-MDT0000 complete [ 4337.132470] LustreError: 144087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4337.168343] LustreError: 144087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4337.374901] LustreError: 142856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077350 with bad export cookie 13320509036146867486 [ 4337.377414] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4337.392505] LustreError: 142856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4338.002925] Lustre: server umount lustre-MDT0001 complete [ 4366.416645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4370.050882] Key type lgssc unregistered [ 4370.561372] LNet: 148725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4370.595432] LNetError: 148725:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4370.623378] LNet: Removed LNI 192.168.203.155@tcp [ 4372.230748] Key type .llcrypt unregistered [ 4372.232562] Key type ._llcrypt unregistered [ 4413.623588] Key type ._llcrypt registered [ 4413.626456] Key type .llcrypt registered [ 4413.738862] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4431.186632] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4432.200461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4432.389083] alg: No test for adler32 (adler32-zlib) [ 4433.477619] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 4433.849707] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4435.615242] Key type lgssc registered [ 4437.144714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4447.844802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4456.774369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4464.394318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4471.634784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4479.223672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4479.289441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4479.313475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.656360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4480.704944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4480.819619] Lustre: lustre-MDT0000: new disk, initializing [ 4480.959810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.019134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4485.635462] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4497.259848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.347456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4497.458333] Lustre: 152677:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4497.558770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4497.560757] Lustre: Skipped 1 previous similar message [ 4497.699818] Lustre: lustre-MDT0001: new disk, initializing [ 4497.831336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4497.861233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4497.874712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4502.554677] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4509.385371] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4512.759796] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4518.927767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4519.030830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4519.226818] Lustre: lustre-OST0000: new disk, initializing [ 4519.230208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4519.300970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.579978] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4525.702444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4525.714938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4525.825040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4532.314707] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4536.786458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4536.988822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4541.017436] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4541.278805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4546.528476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4546.535730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.548257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4549.709115] LustreError: 155618:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4549.917213] Lustre: server umount lustre-OST0000 complete [ 4554.208242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4554.211074] LustreError: Skipped 1 previous similar message [ 4554.220120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.231350] Lustre: Skipped 1 previous similar message [ 4554.235752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.238463] Lustre: Skipped 1 previous similar message [ 4556.769211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.774772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4559.478089] LustreError: 155820:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4559.481956] LustreError: 155820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4559.585016] Lustre: server umount lustre-MDT0000 complete [ 4562.861551] LustreError: 152671:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077576 with bad export cookie 4798903640275188728 [ 4562.877012] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4562.997723] LustreError: 156022:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4563.003875] LustreError: 156022:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4563.175513] Lustre: server umount lustre-MDT0001 complete [ 4571.626190] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 05:26:22 (1769077582) [ 4586.127756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4596.547609] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4606.910583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.972842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4618.656082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4625.621736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4633.430601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4633.525337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4634.022931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4634.076563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4634.258480] Lustre: lustre-MDT0000: new disk, initializing [ 4634.420565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4634.454527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4639.209668] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4652.985454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.118707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4653.228388] Lustre: 159016:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.340930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4653.355213] Lustre: Skipped 1 previous similar message [ 4653.509160] Lustre: lustre-MDT0001: new disk, initializing [ 4653.655191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4653.740861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4653.771670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4660.085933] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4667.847520] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4672.065759] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4679.789772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4679.911318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4680.266349] Lustre: lustre-OST0000: new disk, initializing [ 4680.272251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4680.400262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.332785] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4690.097279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4690.116082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4690.503676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4697.659635] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4701.536748] Lustre: Failing over lustre-OST0000 [ 4701.619354] LustreError: 161460:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4701.622381] LustreError: 161460:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4701.790831] Lustre: server umount lustre-OST0000 complete [ 4720.095381] Lustre: 150889:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769077717/real 1769077717] req@0000000075abbebc x1855012153137664/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1769077733 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4720.120145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.473937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4722.170823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.199415] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4723.167095] Lustre: 150889:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769077719/real 1769077719] req@000000004fdbe9c5 x1855012153138048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1769077735 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4723.199516] Lustre: 150889:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4725.161795] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4727.199232] Lustre: 150888:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769077724/real 1769077724] req@00000000168b2dd8 x1855012153139712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769077740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4730.154860] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4730.416103] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4730.417049] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4732.511210] Lustre: 150891:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769077729/real 1769077729] req@0000000002b8cc71 x1855012153140864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769077745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4732.546416] Lustre: 150891:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4736.413895] Lustre: Failing over lustre-MDT0000 [ 4736.750215] LustreError: 162442:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4736.761237] LustreError: 162442:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4737.524950] Lustre: 150890:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769077733/real 1769077733] req@000000004bdf0758 x1855012153141888/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769077749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4737.583508] Lustre: 150890:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4737.644511] Lustre: server umount lustre-MDT0000 complete [ 4740.585032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4740.596677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.611079] LustreError: 160136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4740.653915] Lustre: Skipped 3 previous similar messages [ 4740.674995] LustreError: 160136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4745.689066] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.718726] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4750.816282] LustreError: 159023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4755.941116] Lustre: 150888:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769077753/real 1769077753] req@000000009b44961c x1855012153152384/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769077769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4755.947226] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4755.971312] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4756.011553] LustreError: 159024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4757.666702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.057469] LustreError: 159028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.071455] LustreError: 159028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4766.495988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.569262] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4767.196153] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4771.202344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4771.862959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4771.872528] Lustre: Skipped 1 previous similar message [ 4771.920198] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4771.969793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4782.056843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4782.064101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.075977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4782.081016] Lustre: Skipped 1 previous similar message [ 4782.097067] Lustre: Skipped 2 previous similar messages [ 4786.518853] LustreError: 163331:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4786.523563] LustreError: 163331:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4786.740503] Lustre: server umount lustre-OST0000 complete [ 4792.801315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4792.821696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4798.720704] Lustre: server umount lustre-MDT0000 complete [ 4802.537302] LustreError: 160137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4802.555154] LustreError: 160137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4803.279163] LustreError: 159007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769077816 with bad export cookie 4798903640275190842 [ 4803.290885] LustreError: 159007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4803.292546] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4803.501160] LustreError: 163743:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4803.506466] LustreError: 163743:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4803.746470] Lustre: server umount lustre-MDT0001 complete [ 4815.745871] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4818.111169] Key type lgssc unregistered [ 4818.328806] LNet: 164277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4818.336452] LNetError: 164277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4819.364948] LNet: Removed LNI 192.168.203.155@tcp [ 4820.099773] Key type .llcrypt unregistered [ 4820.102901] Key type ._llcrypt unregistered [ 4855.078966] Key type ._llcrypt registered [ 4855.080937] Key type .llcrypt registered [ 4855.197857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4856.949938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4857.092425] alg: No test for adler32 (adler32-zlib) [ 4858.289552] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 4858.610283] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4860.344110] Key type lgssc registered [ 4861.450956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4869.508676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4869.556127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.227917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4876.665371] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4881.378498] LustreError: 165781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4881.393588] LustreError: 165781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4883.774353] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4886.508450] LustreError: 165781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4887.246030] LustreError: 165809:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4887.253696] LustreError: 165809:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4887.261517] LustreError: 165809:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4887.362199] LustreError: 166332:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4887.499802] Lustre: server umount lustre-MDT0000 complete [ 4902.254849] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4910.635116] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4919.341905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4925.699798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4931.841897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.297529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.802616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.876242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.205614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4945.239917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4945.311502] Lustre: lustre-MDT0000: new disk, initializing [ 4945.390093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.406240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4949.145898] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4958.863900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4958.924461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4958.983043] Lustre: 168974:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.037559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4959.040843] Lustre: Skipped 1 previous similar message [ 4959.089898] Lustre: lustre-MDT0001: new disk, initializing [ 4959.151856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4959.181519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4959.186261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4964.563537] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4971.687519] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4975.617605] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4982.892310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4982.992607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4983.251409] Lustre: lustre-OST0000: new disk, initializing [ 4983.255651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4983.337133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.923512] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4990.005372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4990.017735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4990.135134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4995.788236] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5001.090415] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5002.983497] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:33:34 (1769078014) [ 5230.560453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5230.570077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.581307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.072524] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.101183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.101192] Lustre: Skipped 1 previous similar message [ 5231.136269] Lustre: Skipped 1 previous similar message [ 5234.315606] LustreError: 172497:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5234.323905] LustreError: 172497:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5234.857950] Lustre: server umount lustre-MDT0000 complete [ 5236.197545] LustreError: 168988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.238111] LustreError: 168988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5239.637498] LustreError: 168968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078252 with bad export cookie 11870495699766275173 [ 5239.640090] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5239.645974] LustreError: 168968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5241.312206] LustreError: 168984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.313265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.367889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5241.383088] Lustre: Skipped 1 previous similar message [ 5244.320042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5245.903107] LustreError: 172701:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5245.910271] LustreError: 172701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5246.246684] Lustre: server umount lustre-MDT0001 complete [ 5256.376592] LustreError: 172902:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5256.391150] LustreError: 172902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5256.571938] Lustre: server umount lustre-OST0000 complete [ 5268.936136] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5282.107301] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5294.399208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5300.601200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5307.000741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5313.419873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5320.187881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5320.272691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.586944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5320.658421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5320.748525] Lustre: lustre-MDT0000: new disk, initializing [ 5320.892091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.924754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5325.790399] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5336.223548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5336.292883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5336.374719] Lustre: 175245:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5336.412155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5336.414309] Lustre: Skipped 1 previous similar message [ 5336.468379] Lustre: lustre-MDT0001: new disk, initializing [ 5336.573353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5336.613213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5336.620955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5340.744619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5348.093610] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5351.646192] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5359.515692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5359.560231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5359.787907] Lustre: lustre-OST0000: new disk, initializing [ 5359.799826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5359.967633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.500123] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5367.871413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5367.874645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5367.978768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5374.257396] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5379.727820] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5379.926378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5384.079891] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5384.245857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5388.768132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5388.769689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.772102] LustreError: Skipped 1 previous similar message [ 5388.773792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5388.808140] Lustre: Skipped 1 previous similar message [ 5399.009769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.031045] Lustre: Skipped 4 previous similar messages [ 5400.033545] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5400.067743] LustreError: 178188:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5400.076803] LustreError: 178188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5400.232218] Lustre: server umount lustre-OST0000 complete [ 5408.229157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5408.256222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.521342] LustreError: 178391:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5411.530779] LustreError: 178391:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5411.737114] Lustre: server umount lustre-MDT0000 complete [ 5414.369886] LustreError: 175253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.388096] LustreError: 175253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5417.539283] LustreError: 175239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078430 with bad export cookie 11870495699766434003 [ 5417.545324] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5417.552539] LustreError: 175239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5417.812846] Lustre: server umount lustre-MDT0001 complete [ 5427.253419] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:40:38 (1769078438) [ 5444.127581] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5454.750285] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5467.134896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5474.444968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5481.979738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5488.377385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5495.276206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5495.356461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.645465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5495.671948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5495.746125] Lustre: lustre-MDT0000: new disk, initializing [ 5495.812104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.839179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5501.864432] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5513.863139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5513.935271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5514.089856] Lustre: 181585:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5514.413755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5518.394320] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5528.029337] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5533.236679] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5540.612741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5540.669283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5540.910265] Lustre: lustre-OST0000: new disk, initializing [ 5540.913851] Lustre: Skipped 1 previous similar message [ 5540.926242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5540.933340] Lustre: Skipped 2 previous similar messages [ 5547.615499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5547.622845] Lustre: Skipped 1 previous similar message [ 5547.631810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5547.777809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5549.659344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5559.542244] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5568.484991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5568.486892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.488291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5568.488297] Lustre: Skipped 3 previous similar messages [ 5568.505522] Lustre: Skipped 2 previous similar messages [ 5573.396122] LustreError: 184136:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5573.398789] LustreError: 184136:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5573.532820] Lustre: server umount lustre-OST0000 complete [ 5583.437962] Lustre: server umount lustre-MDT0000 complete [ 5583.845268] LustreError: 182784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.870364] LustreError: 182784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5587.078934] LustreError: 181579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078600 with bad export cookie 11870495699766434885 [ 5587.087142] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5587.101124] LustreError: 181579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5587.359493] Lustre: server umount lustre-MDT0001 complete [ 5595.187386] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:43:26 (1769078606) [ 5613.628927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5624.591861] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5636.697499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5644.537021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5652.223390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5658.962876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5665.035890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5665.077532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.244900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5665.259493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5665.317240] Lustre: lustre-MDT0000: new disk, initializing [ 5665.357382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5665.359994] Lustre: Skipped 2 previous similar messages [ 5665.369124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5669.757844] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5684.704402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.766902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5684.874852] Lustre: 187535:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5685.189763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5690.255698] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5698.132678] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.051357] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5712.407398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5712.487488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5718.843795] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5720.148727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5720.290173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5725.891557] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5735.906789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5735.912694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.918670] LustreError: Skipped 1 previous similar message [ 5735.925908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5735.928464] Lustre: Skipped 3 previous similar messages [ 5735.934789] Lustre: Skipped 3 previous similar messages [ 5739.093108] LustreError: 190086:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5739.105323] LustreError: 190086:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5739.294490] Lustre: server umount lustre-OST0000 complete [ 5749.466689] Lustre: server umount lustre-MDT0000 complete [ 5751.271759] LustreError: 187549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.304638] LustreError: 187549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5752.959250] LustreError: 188323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078766 with bad export cookie 11870495699766435970 [ 5752.968334] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5752.969491] LustreError: 188323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5753.502680] Lustre: server umount lustre-MDT0001 complete [ 5764.389574] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5767.464890] Key type lgssc unregistered [ 5767.809943] LNet: 191021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5767.828074] LNetError: 191021:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5767.842292] LNet: Removed LNI 192.168.203.155@tcp [ 5768.740710] Key type .llcrypt unregistered [ 5768.746917] Key type ._llcrypt unregistered [ 5791.273091] Key type ._llcrypt registered [ 5791.282951] Key type .llcrypt registered [ 5791.443882] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5792.656340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5792.866243] alg: No test for adler32 (adler32-zlib) [ 5793.919289] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 5794.152236] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5795.893482] Key type lgssc registered [ 5797.048708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5804.855565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5804.879789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.516450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.012979] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5815.777333] LustreError: 192419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.791354] LustreError: 192419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5819.127392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5819.565315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5823.924897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5830.427788] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5835.131126] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5843.389309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5843.905745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.082423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5851.323818] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5856.722841] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5880.150278] LustreError: 194689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5880.380376] Lustre: server umount lustre-OST0000 complete [ 5885.925199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5885.930257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.965190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5886.945966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.947587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5886.971048] Lustre: Skipped 1 previous similar message [ 5892.093774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5892.099616] Lustre: Skipped 1 previous similar message [ 5897.184706] Lustre: 191730:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769078894/real 1769078894] req@000000000e04ad0e x1855013579160320/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1769078910 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5897.184864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.211927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.215245] Lustre: Skipped 1 previous similar message [ 5898.207376] Lustre: 191729:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769078895/real 1769078895] req@000000000438fe36 x1855013579161088/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769078911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5900.255322] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5900.411599] LustreError: 194890:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5900.421832] LustreError: 194890:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5900.705759] Lustre: server umount lustre-MDT0000 complete [ 5902.305233] LustreError: 192416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.336885] LustreError: 192416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5903.263136] Lustre: 191728:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769078900/real 1769078900] req@000000001bc92b9e x1855013579162880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769078916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5904.553338] LustreError: 193140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769078917 with bad export cookie 5843777539396149492 [ 5904.560257] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5904.568102] LustreError: 193140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5904.702407] LustreError: 195093:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5904.705833] LustreError: 195093:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5904.896736] Lustre: server umount lustre-MDT0001 complete [ 5915.765390] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5918.333890] Key type lgssc unregistered [ 5918.616620] LNet: 195625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5918.623476] LNetError: 195625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5919.653536] LNet: Removed LNI 192.168.203.155@tcp [ 5920.582679] Key type .llcrypt unregistered [ 5920.588762] Key type ._llcrypt unregistered [ 5948.570576] Key type ._llcrypt registered [ 5948.573756] Key type .llcrypt registered [ 5948.751849] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5950.444597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5950.559094] alg: No test for adler32 (adler32-zlib) [ 5951.576533] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 5951.770662] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5953.431228] Key type lgssc registered [ 5954.557614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5963.402248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5963.443864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.877599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.450774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5975.009069] LustreError: 197026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.037666] LustreError: 197026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5978.789168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5979.218648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5983.626478] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5989.277620] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5993.557119] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6000.683375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6001.028049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6006.815163] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6010.314422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6012.453078] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6019.554342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6019.557629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.572391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6020.581668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.583167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6020.615821] Lustre: Skipped 2 previous similar messages [ 6025.401256] LustreError: 199277:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6025.601856] Lustre: server umount lustre-OST0000 complete [ 6029.909123] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa0208b39f000.ost_server_uuid 50 [ 6031.609994] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6045.664957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6045.680813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.712115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6051.800030] LustreError: 199644:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6051.805979] LustreError: 199644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6052.192054] Lustre: server umount lustre-MDT0000 complete [ 6052.333698] LustreError: 197023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.352683] LustreError: 197023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6056.491786] LustreError: 198649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079069 with bad export cookie 15474558162668425848 [ 6056.498092] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6056.513027] LustreError: 198649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6056.696821] LustreError: 199846:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6056.709597] LustreError: 199846:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6057.192736] Lustre: server umount lustre-MDT0001 complete [ 6067.104347] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:51:18 (1769079078) [ 6074.552939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.963198] LustreError: 200517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6075.019402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.481621] LustreError: 200516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.505574] LustreError: 200516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6080.644875] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6085.602958] LustreError: 200516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.733639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6090.070984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.681470] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6102.623215] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6109.163949] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6117.785666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6118.166176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.247813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6124.620771] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6131.081536] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6139.356637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6139.470523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6139.676329] Lustre: lustre-OST0001: new disk, initializing [ 6139.678581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6139.853264] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6145.636349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6145.648909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6145.750042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6146.044221] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6153.097111] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6156.257770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6156.272165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.272174] Lustre: Skipped 1 previous similar message [ 6156.272675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.272679] Lustre: Skipped 4 previous similar messages [ 6156.293240] LustreError: Skipped 1 previous similar message [ 6161.393254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6161.399583] Lustre: Skipped 3 previous similar messages [ 6161.609853] LustreError: 203723:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6161.616378] LustreError: 203723:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6161.734168] Lustre: server umount lustre-OST0000 complete [ 6163.937615] LustreError: 202115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.785308] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6167.987240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6171.742114] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6171.935440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6181.875650] LustreError: 202115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.901023] LustreError: 202115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6186.977329] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.988371] Lustre: Skipped 1 previous similar message [ 6186.992526] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6186.993742] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6187.000789] LustreError: Skipped 1 previous similar message [ 6187.011306] Lustre: Skipped 1 previous similar message [ 6191.560129] LustreError: 204445:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6191.562850] LustreError: 204445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6191.671437] Lustre: server umount lustre-OST0001 complete [ 6197.218861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.240436] Lustre: Skipped 2 previous similar messages [ 6201.411510] LustreError: 204646:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6201.420217] LustreError: 204646:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6201.682824] Lustre: server umount lustre-MDT0000 complete [ 6205.223363] LustreError: 203028:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079218 with bad export cookie 15474558162668426919 [ 6205.232801] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6205.236412] LustreError: 203028:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6205.603707] Lustre: server umount lustre-MDT0001 complete [ 6211.015242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6214.207445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6217.875843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.300286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6227.387705] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:53:58 (1769079238) [ 6233.591731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6233.953589] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6234.002960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6234.155545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.408741] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6246.597660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6246.762257] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6246.839720] Lustre: 206343:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6250.929543] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6255.328825] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6258.466199] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6264.475296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6264.653254] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6270.190454] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6270.590431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6275.121280] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6281.042394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6281.113339] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6287.437917] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6291.756597] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6295.526938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.538553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.551969] Lustre: Skipped 1 previous similar message [ 6295.558486] Lustre: Skipped 2 previous similar messages [ 6308.319269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6308.370282] LustreError: 209543:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6308.375622] LustreError: 209543:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6308.463585] Lustre: server umount lustre-OST0000 complete [ 6310.888895] LustreError: 207974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.912767] LustreError: 207974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6325.732379] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6329.739476] Lustre: server umount lustre-OST0001 complete [ 6334.432420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6334.436590] LustreError: Skipped 1 previous similar message [ 6334.439516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.458389] Lustre: Skipped 2 previous similar messages [ 6334.464113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6334.469854] Lustre: Skipped 11 previous similar messages [ 6347.231579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6347.340975] LustreError: 209969:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6347.344621] LustreError: 209969:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6347.476727] Lustre: server umount lustre-MDT0000 complete [ 6350.679640] LustreError: 207993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079364 with bad export cookie 15474558162668428326 [ 6350.684256] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.690179] LustreError: 207993:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6351.115940] Lustre: server umount lustre-MDT0001 complete [ 6355.952322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.573320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.349699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6366.968764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.531933] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:56:24 (1769079384) [ 6388.281662] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6396.313932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6404.156570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.480702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6414.322530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6419.634558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.504859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.559759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.724692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6425.744661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6425.798552] Lustre: lustre-MDT0000: new disk, initializing [ 6425.862317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6425.866903] Lustre: Skipped 3 previous similar messages [ 6425.884980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6429.313677] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6437.605967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.673412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6437.739931] Lustre: 213999:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6437.782915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6437.787278] Lustre: Skipped 1 previous similar message [ 6437.873463] Lustre: lustre-MDT0001: new disk, initializing [ 6438.005397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6438.014944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6441.164047] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6446.972761] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6449.749601] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6454.880597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6454.925993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6455.136307] Lustre: lustre-OST0000: new disk, initializing [ 6455.141338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6459.641683] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6461.461833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6461.471671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6461.542722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6465.217357] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6469.535764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6469.772254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6473.386877] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6473.565693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6476.770726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6476.771166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6476.781068] Lustre: Skipped 1 previous similar message [ 6476.788747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6476.791622] LustreError: Skipped 1 previous similar message [ 6476.794581] Lustre: Skipped 6 previous similar messages [ 6481.079251] LustreError: 216934:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6481.082244] LustreError: 216934:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6481.187940] Lustre: server umount lustre-OST0000 complete [ 6490.278226] Lustre: server umount lustre-MDT0000 complete [ 6492.130038] LustreError: 214009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.152310] LustreError: 214009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6493.099789] LustreError: 213989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079506 with bad export cookie 15474558162668429894 [ 6493.106549] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6493.108013] LustreError: 213989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6493.349474] Lustre: server umount lustre-MDT0001 complete [ 6499.845049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.485129] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6510.720232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6514.735947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6518.995133] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6522.100330] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6527.614986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6532.671136] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6537.016798] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6540.128474] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6545.674573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 6548.668839] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6548.803854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6550.496253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6552.273940] Lustre: server umount lustre-OST0000 complete [ 6556.899565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6560.278689] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6587.649983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6587.861828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6587.869953] Lustre: Skipped 5 previous similar messages [ 6589.932124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.960385] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.963685] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6589.976857] Lustre: Skipped 1 previous similar message [ 6592.492481] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6597.226427] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6606.849689] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 6610.400776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.405467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6610.409673] Lustre: Skipped 6 previous similar messages [ 6610.419285] Lustre: Skipped 6 previous similar messages [ 6615.298398] LustreError: 222232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6615.305094] LustreError: 222232:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6615.380981] Lustre: server umount lustre-OST0000 complete [ 6618.596759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6618.607727] LustreError: Skipped 2 previous similar messages [ 6624.399909] Lustre: server umount lustre-MDT0000 complete [ 6625.761745] LustreError: 217809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.771528] LustreError: 217809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6627.023457] LustreError: 217795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079640 with bad export cookie 15474558162668430776 [ 6627.030574] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.213406] Lustre: server umount lustre-MDT0001 complete [ 6632.993551] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:00:44 (1769079644) [ 6637.858908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.831254] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6646.567794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6650.001450] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6653.549772] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6655.705876] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6660.185681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6663.889945] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6666.624453] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6669.587696] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6669.757896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6671.864852] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6673.034944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 6677.756738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6677.824939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6677.893875] Lustre: lustre-OST0001: new disk, initializing [ 6677.896762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6681.861835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6685.224569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6685.233105] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6685.285416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6686.576919] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6689.513144] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6689.666537] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6692.823876] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6692.970714] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6700.633684] Lustre: server umount lustre-OST0001 complete [ 6704.384914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6704.523556] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6707.489034] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6707.635155] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6733.389902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6737.359482] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6738.928457] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6738.935487] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6738.936629] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6741.111450] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6742.173575] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6749.581471] Lustre: server umount lustre-OST0001 complete [ 6769.824194] Lustre: server umount lustre-OST0000 complete [ 6774.757377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6774.761770] LustreError: Skipped 4 previous similar messages [ 6778.504680] Lustre: server umount lustre-MDT0000 complete [ 6780.737657] LustreError: 226330:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079794 with bad export cookie 15474558162668431742 [ 6780.742955] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6780.747254] LustreError: 226330:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6795.231482] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6795.403596] Lustre: server umount lustre-MDT0001 complete [ 6798.814749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6801.094248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6803.589951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6806.366621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6810.634979] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:03:42 (1769079822) [ 6814.977812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.154857] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6815.180918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6818.007909] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6823.217454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6823.281296] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6823.299149] Lustre: 231305:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6826.411524] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6830.126751] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6832.937383] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6838.257441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6838.360887] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6842.411403] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6846.380354] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6852.541555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6852.576777] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6852.681056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6852.686725] Lustre: Skipped 8 previous similar messages [ 6857.025116] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6860.920907] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6861.927277] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6864.745714] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6864.902454] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6867.730347] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6867.890344] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6870.074187] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa02084c91000.ost_server_uuid 50 [ 6871.052831] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa02084c91000.ost_server_uuid in FULL state after 0 sec [ 6871.779449] Lustre: Permanently deactivating lustre-OST0001 [ 6871.788369] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6872.007292] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6874.338029] Lustre: Permanently reactivating lustre-OST0001 [ 6879.752784] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.759630] Lustre: Skipped 10 previous similar messages [ 6879.764457] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6879.767370] Lustre: Skipped 19 previous similar messages [ 6883.011883] LustreError: 235299:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6883.015252] LustreError: 235299:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6883.110597] Lustre: server umount lustre-OST0001 complete [ 6884.835029] LustreError: 232927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.854778] LustreError: 232927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 6900.191127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6900.191145] LustreError: 235304:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 6900.299637] Lustre: server umount lustre-OST0000 complete [ 6920.674745] LustreError: 235509:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0000: unknown param osc.active=1 [ 6926.964834] Lustre: server umount lustre-MDT0000 complete [ 6929.504095] LustreError: 231297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769079942 with bad export cookie 15474558162668433030 [ 6929.508735] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6929.512273] LustreError: 231297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6929.754876] Lustre: server umount lustre-MDT0001 complete [ 6933.579955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6936.242175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.745217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6941.276551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6945.552591] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:05:57 (1769079957) [ 6946.281840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6950.360240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.531635] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6950.548613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6950.554588] Lustre: Skipped 5 previous similar messages [ 6952.831279] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6957.660810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6957.714793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6957.729346] Lustre: 237460:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6960.371458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6963.879763] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.374321] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6970.806095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6970.867314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6970.988313] Lustre: Permanently deactivating lustre-OST0000 [ 6970.996658] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6974.822728] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6977.048754] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6977.056204] Lustre: Skipped 1 previous similar message [ 6978.086803] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6982.365692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6984.937327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6985.882620] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6989.522940] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6990.485316] Lustre: Permanently reactivating lustre-OST0000 [ 6990.496907] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6990.500656] Lustre: Skipped 2 previous similar messages [ 6990.607087] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6990.612711] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6990.626611] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6990.633397] Lustre: Skipped 1 previous similar message [ 6990.660285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6990.676321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7005.838126] Lustre: server umount lustre-OST0001 complete [ 7022.047283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7022.136228] Lustre: server umount lustre-OST0000 complete [ 7030.762804] Lustre: server umount lustre-MDT0000 complete [ 7032.762302] LustreError: 237455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080046 with bad export cookie 15474558162668434801 [ 7032.767496] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7032.771533] LustreError: 237455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7032.953368] Lustre: server umount lustre-MDT0001 complete [ 7038.520827] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7039.963052] Key type lgssc unregistered [ 7040.117778] LNet: 241852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7040.122169] LNetError: 241852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7040.132832] LNet: Removed LNI 192.168.203.155@tcp [ 7040.563665] Key type .llcrypt unregistered [ 7040.565152] Key type ._llcrypt unregistered [ 7052.515546] Key type ._llcrypt registered [ 7052.519204] Key type .llcrypt registered [ 7052.573102] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7053.033434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7053.062506] alg: No test for adler32 (adler32-zlib) [ 7053.997635] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7054.166669] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7055.775291] Key type lgssc registered [ 7056.426561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7058.205430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7061.478230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7061.488540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.736762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7064.567408] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7068.650040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.681528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7068.731739] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7068.745260] Lustre: 243265:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7068.750932] Lustre: Permanently deactivating lustre-MDT0001 [ 7068.757563] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7068.895135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7070.521784] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7070.905542] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7070.915368] LustreError: 243309:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7070.924506] LustreError: 243309:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7070.931549] LustreError: 243309:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 7073.046897] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7074.629924] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7077.766978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7077.969691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7079.013466] LustreError: 245037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.018059] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7079.029384] LustreError: 245037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7080.650853] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7083.729832] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7084.515342] LustreError: 245037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.319245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7087.459590] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.051203] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7089.071583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7089.074247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7090.925671] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7094.059055] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7096.128588] Lustre: Permanently reactivating lustre-MDT0001 [ 7096.139961] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7096.143938] Lustre: Skipped 2 previous similar messages [ 7099.496538] Lustre: lustre-MDT0001: Client 25762c56-4f50-4334-bf79-dc349d16e293 (at 192.168.203.55@tcp) reconnecting [ 7101.951311] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7102.758513] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7105.177740] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7105.312507] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7107.186666] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7107.290937] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7108.607432] Lustre: Permanently deactivating lustre-MDT0001 [ 7108.617616] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7108.621441] Lustre: Skipped 2 previous similar messages [ 7113.749390] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7119.842592] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.843676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.847628] Lustre: Skipped 2 previous similar messages [ 7119.850356] Lustre: Skipped 1 previous similar message [ 7123.329305] LustreError: 247391:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7123.359188] Lustre: server umount lustre-MDT0000 complete [ 7124.943282] LustreError: 244004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080138 with bad export cookie 7830746028895382419 [ 7124.947066] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7124.947424] LustreError: 244004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7124.961538] LustreError: 243274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.969148] LustreError: 243274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7125.041486] LustreError: 247594:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7125.044902] LustreError: 247594:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7125.134534] Lustre: server umount lustre-MDT0001 complete [ 7132.845108] LustreError: 247795:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7132.848475] LustreError: 247795:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7132.867409] Lustre: server umount lustre-OST0000 complete [ 7137.619150] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:09:09 (1769080149) [ 7142.171231] LustreError: 248799:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7142.178308] LustreError: 248799:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7142.230978] Lustre: server umount lustre-OST0001 complete [ 7144.916868] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7149.350529] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7154.038256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7157.664270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7160.516298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7169.209249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7174.362412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7174.438491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.715197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7174.720271] Lustre: Skipped 2 previous similar messages [ 7174.743142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7174.798636] Lustre: lustre-MDT0000: new disk, initializing [ 7174.873279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7174.889636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7177.790477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7185.240241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7185.300365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7185.378130] Lustre: 250983:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7185.427696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7185.430356] Lustre: Skipped 1 previous similar message [ 7185.481371] Lustre: lustre-MDT0001: new disk, initializing [ 7185.557595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7185.607311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7185.619845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7188.585503] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7194.037367] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7196.516794] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7200.696868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7200.751250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7200.890956] Lustre: lustre-OST0000: new disk, initializing [ 7200.893950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7204.522774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7206.953587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7206.959109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7207.016573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7209.377514] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7210.589615] LustreError: 250990:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7215.948049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7215.987397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7216.070306] Lustre: lustre-OST0001: new disk, initializing [ 7216.073171] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7216.116357] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7216.121822] Lustre: Skipped 1 previous similar message [ 7216.296117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7216.312401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7216.378103] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7219.661798] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7220.623057] LustreError: 250990:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7221.658724] LustreError: 250990:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7224.571539] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7225.394068] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7231.663111] LustreError: 250990:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7236.585991] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 7236.591515] Lustre: Skipped 2 previous similar messages [ 7237.090269] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7237.090792] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.094037] LustreError: Skipped 1 previous similar message [ 7237.103791] Lustre: Skipped 2 previous similar messages [ 7238.717606] LustreError: 254478:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7238.722501] LustreError: 254478:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7238.757983] Lustre: server umount lustre-OST0001 complete [ 7242.220927] LustreError: 252715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7242.231331] LustreError: 252715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7247.337344] LustreError: 252714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.347856] LustreError: 252714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7252.451759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.454256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.459759] Lustre: Skipped 1 previous similar message [ 7252.463906] Lustre: Skipped 3 previous similar messages [ 7257.570226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7257.570660] LustreError: 252714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.573954] Lustre: Skipped 1 previous similar message [ 7257.585829] LustreError: 252714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7258.554265] LustreError: 254703:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7258.558987] LustreError: 254703:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7258.623563] Lustre: server umount lustre-OST0000 complete [ 7262.689050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.690195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.691740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7262.699182] Lustre: Skipped 1 previous similar message [ 7266.864876] Lustre: server umount lustre-MDT0000 complete [ 7268.769765] LustreError: 250974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080282 with bad export cookie 7830746028895385226 [ 7268.778141] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7283.167297] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7283.259954] LustreError: 255106:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7283.265951] LustreError: 255106:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7283.362991] Lustre: server umount lustre-MDT0001 complete [ 7288.262248] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7289.518364] Key type lgssc unregistered [ 7289.683521] LNet: 255639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7289.688309] LNetError: 255639:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7289.700408] LNet: Removed LNI 192.168.203.155@tcp [ 7290.111542] Key type .llcrypt unregistered [ 7290.112881] Key type ._llcrypt unregistered [ 7304.380184] Key type ._llcrypt registered [ 7304.381670] Key type .llcrypt registered [ 7304.432110] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7312.860902] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7313.323392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7313.429441] alg: No test for adler32 (adler32-zlib) [ 7314.342349] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7314.462605] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7316.055151] Key type lgssc registered [ 7316.580832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7321.282180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7324.996214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7328.090979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7331.551863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7335.197055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7335.230420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7335.242743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.372650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7336.391627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7336.439131] Lustre: lustre-MDT0000: new disk, initializing [ 7336.507502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7336.522308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7338.666605] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7344.495304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7344.537222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7344.588739] Lustre: 259562:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7344.625636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7344.627513] Lustre: Skipped 1 previous similar message [ 7344.669155] Lustre: lustre-MDT0001: new disk, initializing [ 7344.730335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7344.756567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7344.761683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7346.955368] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7351.601675] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7353.508128] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7356.441491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7356.472172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7356.587407] Lustre: lustre-OST0000: new disk, initializing [ 7356.590211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7356.645615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.006671] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7362.595468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7362.604443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7362.661799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7363.034793] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7365.059256] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7365.167462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7366.818663] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7366.919337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7372.769279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7372.770148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.772343] LustreError: Skipped 1 previous similar message [ 7372.778406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7372.782711] Lustre: Skipped 1 previous similar message [ 7373.941099] LustreError: 262499:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7373.999787] Lustre: server umount lustre-OST0000 complete [ 7377.889889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.891519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.895681] Lustre: Skipped 2 previous similar messages [ 7377.901592] Lustre: Skipped 1 previous similar message [ 7381.766657] LustreError: 262701:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7381.769932] LustreError: 262701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7381.801878] Lustre: server umount lustre-MDT0000 complete [ 7383.425614] LustreError: 259555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080396 with bad export cookie 1052048738093682066 [ 7383.429179] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7383.431103] LustreError: 259555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7383.503898] LustreError: 262902:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7383.506888] LustreError: 262902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7383.577264] Lustre: server umount lustre-MDT0001 complete [ 7387.040655] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 06:13:19 (1769080399) [ 7389.954922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.184884] LustreError: 263573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.194097] LustreError: 263573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7390.231353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7391.917328] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7395.296558] LustreError: 263573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7396.077563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7396.272594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7398.349720] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7400.985308] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7402.600721] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7405.809283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7405.985474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.433909] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7410.928451] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7456.226108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7456.227744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7456.231109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7456.234639] Lustre: Skipped 1 previous similar message [ 7461.345501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7461.348050] Lustre: Skipped 2 previous similar messages [ 7461.614914] LustreError: 265893:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7461.618828] LustreError: 265893:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7461.676960] Lustre: server umount lustre-OST0000 complete [ 7463.711323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7469.635141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7469.808510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.523366] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7475.032352] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7486.183196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7486.188701] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7486.191214] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7486.196522] Lustre: Skipped 1 previous similar message [ 7512.032838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7512.034661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.037463] LustreError: Skipped 1 previous similar message [ 7512.044251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.048346] Lustre: Skipped 1 previous similar message [ 7515.057084] LustreError: 267559:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7515.061638] LustreError: 267559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7515.119956] Lustre: server umount lustre-OST0000 complete [ 7517.152854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.158684] Lustre: Skipped 2 previous similar messages [ 7522.272121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7522.275144] Lustre: Skipped 3 previous similar messages [ 7522.789920] Lustre: server umount lustre-MDT0000 complete [ 7524.511541] LustreError: 263553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080537 with bad export cookie 1052048738093682934 [ 7524.514945] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7524.517285] LustreError: 263553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7524.593080] LustreError: 267962:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7524.596286] LustreError: 267962:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7524.671682] Lustre: server umount lustre-MDT0001 complete [ 7528.941623] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7530.175985] Key type lgssc unregistered [ 7530.325656] LNet: 268493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7530.329276] LNetError: 268493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7530.339320] LNet: Removed LNI 192.168.203.155@tcp [ 7530.743935] Key type .llcrypt unregistered [ 7530.745187] Key type ._llcrypt unregistered [ 7541.766205] Key type ._llcrypt registered [ 7541.770914] Key type .llcrypt registered [ 7541.833300] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7542.300741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7542.326061] alg: No test for adler32 (adler32-zlib) [ 7543.249230] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7543.360415] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7544.967211] Key type lgssc registered [ 7545.428086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7550.517211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7550.528774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.784667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.404367] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7557.396594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7557.559476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7559.186811] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7561.430269] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7562.801827] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7565.786048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7565.941992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7568.147834] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7570.360395] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7577.195158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7592.930745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.930751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7592.931430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7592.934638] Lustre: Skipped 1 previous similar message [ 7595.886194] LustreError: 272910:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7595.930589] Lustre: server umount lustre-OST0000 complete [ 7598.048826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7598.050645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.054189] Lustre: Skipped 1 previous similar message [ 7598.057035] Lustre: Skipped 1 previous similar message [ 7603.168470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7603.172286] Lustre: Skipped 2 previous similar messages [ 7603.519471] LustreError: 273112:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7603.522295] LustreError: 273112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7603.559751] Lustre: server umount lustre-MDT0000 complete [ 7605.134466] LustreError: 271489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080618 with bad export cookie 5766875556147662774 [ 7605.137529] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7605.139394] LustreError: 271489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7605.224385] LustreError: 273315:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7605.226740] LustreError: 273315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7605.302316] Lustre: server umount lustre-MDT0001 complete [ 7609.811229] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7611.063142] Key type lgssc unregistered [ 7611.238885] LNet: 273846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7611.243534] LNetError: 273846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7611.256406] LNet: Removed LNI 192.168.203.155@tcp [ 7611.661205] Key type .llcrypt unregistered [ 7611.662834] Key type ._llcrypt unregistered [ 7620.034924] Key type ._llcrypt registered [ 7620.036112] Key type .llcrypt registered [ 7620.078485] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7620.462575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7620.525525] alg: No test for adler32 (adler32-zlib) [ 7621.413600] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7621.513482] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7623.111211] Key type lgssc registered [ 7623.573216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7627.085738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7627.094757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.394095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.155566] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7634.334960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7634.542357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7636.403649] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7638.943168] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7640.356137] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7643.057030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7643.200030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.481510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7647.783598] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7653.487905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7664.097598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7664.100662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7664.109985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.118530] Lustre: Skipped 1 previous similar message [ 7669.173103] LustreError: 277436:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7669.235262] Lustre: server umount lustre-OST0000 complete [ 7674.337244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.338742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7674.351109] Lustre: Skipped 2 previous similar messages [ 7679.456120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.458319] Lustre: Skipped 2 previous similar messages [ 7684.576894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7684.580221] Lustre: Skipped 1 previous similar message [ 7685.087127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7685.131120] LustreError: 277637:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7685.134477] LustreError: 277637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7685.171367] Lustre: server umount lustre-MDT0000 complete [ 7686.760600] LustreError: 275005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080700 with bad export cookie 12151591513429906538 [ 7686.761849] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.766061] LustreError: 275005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7686.844570] LustreError: 277841:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7686.847921] LustreError: 277841:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7686.913091] Lustre: server umount lustre-MDT0001 complete [ 7690.958481] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7692.128084] Key type lgssc unregistered [ 7692.268426] LNet: 278372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7692.276471] LNetError: 278372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7692.283415] LNet: Removed LNI 192.168.203.155@tcp [ 7692.640699] Key type .llcrypt unregistered [ 7692.642528] Key type ._llcrypt unregistered [ 7703.232046] Key type ._llcrypt registered [ 7703.233493] Key type .llcrypt registered [ 7703.294297] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7703.774738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7703.783949] alg: No test for adler32 (adler32-zlib) [ 7704.681758] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7704.783858] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7706.375221] Key type lgssc registered [ 7706.861726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7710.238764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7710.249409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.478781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.136945] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7716.935185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7717.097414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7718.683871] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7720.816981] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7722.362135] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7725.337262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7725.525773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.810810] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7729.920631] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7735.785035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7751.139040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.139674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7751.140478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7751.144960] Lustre: Skipped 1 previous similar message [ 7756.258853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7756.261664] Lustre: Skipped 2 previous similar messages [ 7757.110163] LustreError: 282801:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7757.173240] Lustre: server umount lustre-OST0000 complete [ 7761.377217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.377764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.382375] Lustre: Skipped 1 previous similar message [ 7766.496879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.501091] Lustre: Skipped 2 previous similar messages [ 7771.618254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.621420] Lustre: Skipped 1 previous similar message [ 7773.151633] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7773.211827] LustreError: 283002:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7773.216015] LustreError: 283002:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7773.254932] Lustre: server umount lustre-MDT0000 complete [ 7776.332510] LustreError: 283004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080789 with bad export cookie 7230889686034475279 [ 7776.334950] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7776.336269] LustreError: 283004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7776.420287] LustreError: 283207:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7776.424219] LustreError: 283207:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7776.499361] Lustre: server umount lustre-MDT0001 complete [ 7781.447464] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7782.730289] Key type lgssc unregistered [ 7782.881751] LNet: 283738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.888673] LNetError: 283738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.899786] LNet: Removed LNI 192.168.203.155@tcp [ 7783.276675] Key type .llcrypt unregistered [ 7783.277938] Key type ._llcrypt unregistered [ 7791.527892] Key type ._llcrypt registered [ 7791.529338] Key type .llcrypt registered [ 7791.576805] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7791.923744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7791.978533] alg: No test for adler32 (adler32-zlib) [ 7792.869393] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7792.989757] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7794.607149] Key type lgssc registered [ 7795.140636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7798.369021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7798.379422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7799.632450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7801.302975] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7804.856271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7805.004693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7806.575878] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7808.826096] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7810.507324] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7813.567276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7813.720529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.039446] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7818.093996] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7823.977414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7834.597149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.597730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7834.598188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7834.603418] Lustre: Skipped 1 previous similar message [ 7838.961244] LustreError: 287365:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7839.016205] Lustre: server umount lustre-OST0000 complete [ 7840.736083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7840.739039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.744826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7840.748926] Lustre: Skipped 1 previous similar message [ 7844.833229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7844.833469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.954327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7849.958057] Lustre: Skipped 2 previous similar messages [ 7855.071298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7855.072716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7855.080309] Lustre: Skipped 1 previous similar message [ 7855.117993] LustreError: 287566:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7855.120826] LustreError: 287566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7855.153042] Lustre: server umount lustre-MDT0000 complete [ 7856.799463] LustreError: 284897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080870 with bad export cookie 17456099485517786877 [ 7856.801164] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7856.806507] LustreError: 284897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7856.891169] LustreError: 287768:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7856.894524] LustreError: 287768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7856.964200] Lustre: server umount lustre-MDT0001 complete [ 7861.518102] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7862.806839] Key type lgssc unregistered [ 7862.946862] LNet: 288300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7862.951703] LNetError: 288300:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7862.962692] LNet: Removed LNI 192.168.203.155@tcp [ 7863.366103] Key type .llcrypt unregistered [ 7863.368156] Key type ._llcrypt unregistered [ 7869.082639] Key type ._llcrypt registered [ 7869.083778] Key type .llcrypt registered [ 7869.133741] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7876.355728] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7883.505949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7883.818195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7883.838907] alg: No test for adler32 (adler32-zlib) [ 7884.741543] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 7884.856992] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7886.487192] Key type lgssc registered [ 7886.991231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7890.749419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7893.807205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.709461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7899.556030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.399049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.425586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7902.433187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.534482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7903.551377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7903.591077] Lustre: lustre-MDT0000: new disk, initializing [ 7903.621019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.629233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7905.301717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7910.645398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7910.682592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7910.727181] Lustre: 291785:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7910.758749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7910.761205] Lustre: Skipped 1 previous similar message [ 7910.806230] Lustre: lustre-MDT0001: new disk, initializing [ 7910.851358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7910.867319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7910.871856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7912.450715] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7916.097169] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7917.657985] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7920.412308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7920.443564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7920.545107] Lustre: lustre-OST0000: new disk, initializing [ 7920.547678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7920.582961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7922.696447] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7926.169401] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7927.935413] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7930.367353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7930.372395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7930.408862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7931.105830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7932.692845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7932.772944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7935.457583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7935.458536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.460748] LustreError: Skipped 1 previous similar message [ 7935.461400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7935.464857] Lustre: Skipped 1 previous similar message [ 7939.564437] LustreError: 294745:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7939.607598] Lustre: server umount lustre-OST0000 complete [ 7941.599931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7941.602518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.607751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7941.609712] Lustre: Skipped 1 previous similar message [ 7945.696507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7945.696848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.135058] LustreError: 294946:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7947.137324] LustreError: 294946:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7947.167026] Lustre: server umount lustre-MDT0000 complete [ 7948.471404] LustreError: 291778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769080961 with bad export cookie 14826474934918573378 [ 7948.474322] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7948.476069] LustreError: 291778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7948.538835] LustreError: 295148:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7948.540742] LustreError: 295148:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7948.595304] Lustre: server umount lustre-MDT0001 complete [ 7951.291636] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:22:43 (1769080963) [ 7953.612764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.780255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7955.086760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7958.064918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7958.192723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7959.584614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7961.407940] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7962.608733] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7965.067093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7967.132664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7969.024767] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7988.704091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7988.707339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.712904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7988.715300] Lustre: Skipped 1 previous similar message [ 7993.709063] LustreError: 298087:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7993.712238] LustreError: 298087:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7993.753968] Lustre: server umount lustre-OST0000 complete [ 7997.922574] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7997.925977] LustreError: Skipped 1 previous similar message [ 7997.927855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.932560] Lustre: Skipped 1 previous similar message [ 7997.936770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.940356] Lustre: Skipped 2 previous similar messages [ 8001.918108] LustreError: 298289:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8001.920814] LustreError: 298289:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8001.947747] Lustre: server umount lustre-MDT0000 complete [ 8003.303381] LustreError: 296536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081016 with bad export cookie 14826474934918574246 [ 8003.306109] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8003.306329] LustreError: 296539:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 8003.306459] LustreError: 296539:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cd5cfbff x1855015771764352/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8003.307972] LustreError: 296536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8003.458774] Lustre: server umount lustre-MDT0001 complete [ 8007.097532] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8008.256123] Key type lgssc unregistered [ 8008.386531] LNet: 299021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8008.389445] LNetError: 299021:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8008.399731] LNet: Removed LNI 192.168.203.155@tcp [ 8008.725849] Key type .llcrypt unregistered [ 8008.727443] Key type ._llcrypt unregistered [ 8018.101512] Key type ._llcrypt registered [ 8018.102725] Key type .llcrypt registered [ 8018.141715] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8024.554404] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8024.981806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8025.014537] alg: No test for adler32 (adler32-zlib) [ 8025.893487] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 8025.989991] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8027.583507] Key type lgssc registered [ 8028.018813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8031.316502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8033.806215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8036.111042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8038.700262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8041.498377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8045.554805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8054.217505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8054.286871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8054.301529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.625599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8055.688242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8055.826900] Lustre: lustre-MDT0000: new disk, initializing [ 8055.927378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8055.939976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8059.475168] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8069.055023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8069.125564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8069.231574] Lustre: 302802:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8069.283818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8069.287715] Lustre: Skipped 1 previous similar message [ 8069.348949] Lustre: lustre-MDT0001: new disk, initializing [ 8069.460362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8069.501059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8069.512612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8073.838887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8082.638802] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8087.032254] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8095.011300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8095.110945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8095.688723] Lustre: lustre-OST2710: new disk, initializing [ 8095.701519] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8095.913817] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8104.569594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8104.590951] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8104.785349] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8105.214270] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8113.039347] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8118.917551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8118.988418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8119.144616] Lustre: lustre-OST03e8: new disk, initializing [ 8119.149192] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8119.244731] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8123.658186] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8126.039160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8126.044183] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8126.137136] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8127.977467] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8133.903430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8134.084550] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8137.352153] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8137.549906] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8140.483189] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8140.624829] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8143.362799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8143.520765] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8146.926722] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.933245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8146.939727] Lustre: Skipped 1 previous similar message [ 8146.949201] Lustre: Skipped 2 previous similar messages [ 8151.520088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8151.526205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8151.534119] Lustre: Skipped 1 previous similar message [ 8151.537985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8152.709678] LustreError: 307147:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8154.787848] LustreError: 307147:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 8154.801416] LustreError: 307147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8154.850205] Lustre: server umount lustre-MDT0000 complete [ 8157.577193] LustreError: 302793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081170 with bad export cookie 15537547684785468031 [ 8157.578505] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.589097] LustreError: 302793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8157.699453] LustreError: 307349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8157.703602] LustreError: 307349:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8157.837914] Lustre: server umount lustre-MDT0001 complete [ 8171.538223] LustreError: 307549:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8171.546745] LustreError: 307549:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8171.578261] Lustre: server umount lustre-OST2710 complete [ 8184.540043] LustreError: 307751:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8184.550366] LustreError: 307751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8184.754633] Lustre: server umount lustre-OST03e8 complete [ 8199.645799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8207.601953] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8215.114628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8220.128282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8224.614020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8229.279305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8234.397950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8234.437945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.575563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8234.600506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8234.660972] Lustre: lustre-MDT0000: new disk, initializing [ 8234.707931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.715437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8237.792067] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8246.146633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8246.221407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8246.288057] Lustre: 310549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8246.319723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8246.321708] Lustre: Skipped 1 previous similar message [ 8246.358800] Lustre: lustre-MDT0001: new disk, initializing [ 8246.414988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8246.459502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8246.475440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8249.838936] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8256.109909] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8258.570681] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8263.905903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8263.969792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8264.136927] Lustre: lustre-OST0000: new disk, initializing [ 8264.144309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8264.212439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.542820] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8269.888522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8269.908717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8270.008758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8274.447387] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8277.783202] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8278.002377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8281.639312] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8281.826909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8285.664780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8285.669262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.672833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.687255] Lustre: Skipped 1 previous similar message [ 8285.692363] Lustre: Skipped 5 previous similar messages [ 8289.402168] LustreError: 313488:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8289.418182] LustreError: 313488:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8289.504274] Lustre: server umount lustre-OST0000 complete [ 8292.831976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8292.836796] LustreError: Skipped 1 previous similar message [ 8292.840253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.848686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8298.639235] Lustre: server umount lustre-MDT0000 complete [ 8301.024305] LustreError: 310560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8301.050227] LustreError: 310560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8301.779618] LustreError: 310540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081315 with bad export cookie 15537547684785469704 [ 8301.782701] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8301.787443] LustreError: 310540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8302.118418] Lustre: server umount lustre-MDT0001 complete [ 8308.935856] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:28:40 (1769081320) [ 8310.134765] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8311.553726] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:28:43 (1769081323) [ 8314.301882] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8320.737611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8323.963387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8327.192651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8330.363051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8331.227417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8337.349871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.570106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8337.591780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8337.745046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8340.751396] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8346.555980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8346.621697] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8346.650208] Lustre: 316022:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8350.206909] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8354.182891] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8356.870329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8361.464737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8361.551282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8361.704907] Lustre: Denying initial registration attempt from nid 192.168.203.155@tcp, specified as failover [ 8361.710360] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8361.720053] LustreError: 317637:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8361.725225] LustreError: 317637:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8361.730144] LustreError: 317637:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8361.771677] Lustre: server umount lustre-OST0000 complete [ 8361.773541] LustreError: 317637:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8366.564476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8366.568851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8366.581376] Lustre: Skipped 2 previous similar messages [ 8366.607362] Lustre: Skipped 3 previous similar messages [ 8372.382122] LustreError: 318005:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8372.385042] LustreError: 318005:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8372.435945] Lustre: server umount lustre-MDT0000 complete [ 8375.215171] LustreError: 316013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081388 with bad export cookie 15537547684785470593 [ 8375.218571] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8375.224606] LustreError: 316013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8375.516762] Lustre: server umount lustre-MDT0001 complete [ 8382.903784] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8384.759298] Key type lgssc unregistered [ 8384.974861] LNet: 318738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8384.983539] LNetError: 318738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8385.004376] LNet: Removed LNI 192.168.203.155@tcp [ 8385.757410] Key type .llcrypt unregistered [ 8385.760608] Key type ._llcrypt unregistered [ 8393.606205] Key type ._llcrypt registered [ 8393.609174] Key type .llcrypt registered [ 8393.699100] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8394.545367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8394.632943] alg: No test for adler32 (adler32-zlib) [ 8395.634101] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 8395.871317] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8397.495238] Key type lgssc registered [ 8398.354773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8402.946308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8405.595039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8408.543943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8411.591548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8412.513319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8430.539324] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8439.770718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8439.782388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.958798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8440.974581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8441.077915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.121145] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8450.961283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8451.019948] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8451.039657] Lustre: 321496:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8451.213475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8454.440549] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8459.025455] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8462.192160] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8467.742073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8467.830096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8468.000182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8468.174962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8472.714957] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8477.205885] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8480.736890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8480.743100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8480.743235] LustreError: Skipped 1 previous similar message [ 8480.745414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8480.765309] Lustre: Skipped 1 previous similar message [ 8485.110505] LustreError: 323780:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8485.256451] Lustre: server umount lustre-OST0000 complete [ 8490.977281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8490.980180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8490.986689] Lustre: Skipped 1 previous similar message [ 8490.995216] Lustre: Skipped 1 previous similar message [ 8494.385408] LustreError: 323981:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8494.388153] LustreError: 323981:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8494.508171] Lustre: server umount lustre-MDT0000 complete [ 8497.298206] LustreError: 321487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081510 with bad export cookie 2173754129757393433 [ 8497.307932] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8497.308571] LustreError: 321487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8497.431967] LustreError: 324182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8497.440116] LustreError: 324182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8497.634551] Lustre: server umount lustre-MDT0001 complete [ 8505.493190] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8507.368264] Key type lgssc unregistered [ 8507.606730] LNet: 324714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8507.610262] LNetError: 324714:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8507.629553] LNet: Removed LNI 192.168.203.155@tcp [ 8508.159327] Key type .llcrypt unregistered [ 8508.161349] Key type ._llcrypt unregistered [ 8524.362971] Key type ._llcrypt registered [ 8524.364517] Key type .llcrypt registered [ 8524.450136] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8525.290772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8525.305745] alg: No test for adler32 (adler32-zlib) [ 8526.313956] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 8526.477418] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8528.111764] Key type lgssc registered [ 8528.951777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8534.856924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8534.870782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8536.233134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.731394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8545.763705] LustreError: 326092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8545.774295] LustreError: 326092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8546.531829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8546.764155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8549.787271] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8553.711193] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8555.863267] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8559.975262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8560.181813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.835642] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8567.243946] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8580.070862] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 8581.199557] LustreError: 328359:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8581.273714] Lustre: server umount lustre-MDT0000 complete [ 8582.626737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8582.631144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.631857] LustreError: 326085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.638169] Lustre: Skipped 1 previous similar message [ 8582.661019] LustreError: 326085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8583.223810] LustreError: 327713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081596 with bad export cookie 12285386312191154352 [ 8583.226542] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8583.228122] LustreError: 327713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8585.178204] LustreError: 326085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.190419] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 8587.750680] LustreError: 326086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8587.750934] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.767060] Lustre: Skipped 1 previous similar message [ 8587.770988] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8589.389203] LustreError: 328559:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8589.394571] LustreError: 328559:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8589.491513] Lustre: server umount lustre-MDT0001 complete [ 8592.557126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8596.942214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.227129] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xaa7e75c29a99b4b0 [ 8608.231356] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 8608.352944] LustreError: 327702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.360350] LustreError: 327702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8608.398608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.429722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8609.889837] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8613.403117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8613.535668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8614.565346] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8615.075097] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8617.229601] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8617.888713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8618.662091] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8619.302912] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8627.038551] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 7 sec [ 8629.219898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.220172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.222265] LustreError: Skipped 1 previous similar message [ 8629.222797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.227219] Lustre: Skipped 1 previous similar message [ 8629.228699] Lustre: Skipped 1 previous similar message [ 8634.337134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.339930] Lustre: Skipped 2 previous similar messages [ 8634.672977] LustreError: 330866:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8634.675630] LustreError: 330866:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8634.717595] Lustre: server umount lustre-OST0000 complete [ 8639.456719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8639.457101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.463543] Lustre: Skipped 1 previous similar message [ 8642.367472] LustreError: 331067:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8642.370099] LustreError: 331067:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8642.396834] Lustre: server umount lustre-MDT0000 complete [ 8643.655427] LustreError: 326073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081656 with bad export cookie 12285386312191161520 [ 8643.656096] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8643.659690] LustreError: 326073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8643.773755] Lustre: server umount lustre-MDT0001 complete [ 8647.267435] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8648.407941] Key type lgssc unregistered [ 8648.562597] LNet: 331800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8648.565660] LNetError: 331800:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8648.579429] LNet: Removed LNI 192.168.203.155@tcp [ 8648.960145] Key type .llcrypt unregistered [ 8648.961368] Key type ._llcrypt unregistered [ 8659.109273] Key type ._llcrypt registered [ 8659.110665] Key type .llcrypt registered [ 8659.153612] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8659.469155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8659.475852] alg: No test for adler32 (adler32-zlib) [ 8660.332557] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 8660.418257] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8662.007227] Key type lgssc registered [ 8662.422348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8665.144282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8665.150346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.303028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.676572] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8671.040439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8671.230048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8672.826858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8674.814155] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8676.041079] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8681.312941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.319841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.322258] Lustre: Skipped 1 previous similar message [ 8683.321940] LustreError: 334567:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8683.344854] Lustre: server umount lustre-MDT0000 complete [ 8684.799803] LustreError: 333156:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081698 with bad export cookie 16709654714268707388 [ 8684.802850] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8684.803852] LustreError: 333156:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8684.883526] LustreError: 334769:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8684.886266] LustreError: 334769:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8684.947891] Lustre: server umount lustre-MDT0001 complete [ 8690.676677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.783973] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8690.798710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8690.875815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8692.461883] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8695.811218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8695.853903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8695.864873] Lustre: 335537:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8695.965452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8697.355636] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8699.341103] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8700.609803] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8703.218341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8703.293112] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8703.298363] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8703.302437] LustreError: 337198:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 8703.305773] LustreError: 337198:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8703.308865] LustreError: 337198:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8703.331538] Lustre: server umount lustre-OST0000 complete [ 8703.333294] LustreError: 337198:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8706.340045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8706.412732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8706.480070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.603381] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8710.974765] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8714.291690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8714.991197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8715.030252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8715.076333] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8715.115489] Lustre: lustre-OST0001: new disk, initializing [ 8715.117873] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8715.165188] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8718.742857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8720.823909] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8720.925342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8720.930232] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8720.970182] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8725.991600] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8725.992208] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.992990] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8725.994619] LustreError: Skipped 1 previous similar message [ 8725.999847] Lustre: Skipped 2 previous similar messages [ 8728.109115] LustreError: 339318:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8728.111230] LustreError: 339318:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8728.133509] Lustre: server umount lustre-OST0001 complete [ 8729.568152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8729.568320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.571948] LustreError: Skipped 1 previous similar message [ 8729.577855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8729.581687] Lustre: Skipped 2 previous similar messages [ 8731.110181] LustreError: 337666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8731.115817] LustreError: 337666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8735.599120] LustreError: 339519:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8735.601441] LustreError: 339519:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8735.640126] Lustre: server umount lustre-OST0000 complete [ 8737.247871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8737.250352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.254586] Lustre: Skipped 1 previous similar message [ 8737.256529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8737.258965] Lustre: Skipped 2 previous similar messages [ 8741.345292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.345424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8741.354734] Lustre: Skipped 1 previous similar message [ 8743.181733] LustreError: 339720:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8743.184182] LustreError: 339720:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8743.217523] Lustre: server umount lustre-MDT0000 complete [ 8745.187470] LustreError: 335527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081758 with bad export cookie 16709654714268707927 [ 8745.193028] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8745.195492] LustreError: 335527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8745.316876] Lustre: server umount lustre-MDT0001 complete [ 8749.432083] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8750.423323] Key type lgssc unregistered [ 8750.546472] LNet: 340452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8750.548933] LNetError: 340452:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8750.555284] LNet: Removed LNI 192.168.203.155@tcp [ 8750.856683] Key type .llcrypt unregistered [ 8750.857933] Key type ._llcrypt unregistered [ 8763.835032] Key type ._llcrypt registered [ 8763.836147] Key type .llcrypt registered [ 8763.881758] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8771.057239] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8771.482754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8771.535611] alg: No test for adler32 (adler32-zlib) [ 8772.430374] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 8772.533401] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8774.127152] Key type lgssc registered [ 8774.762644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8778.714912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8781.280532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8783.689449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8786.758240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8789.637088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8789.676822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8789.688245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8790.778181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8790.790545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8790.828706] Lustre: lustre-MDT0000: new disk, initializing [ 8790.860464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8790.868331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8792.335133] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8797.096564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8797.120341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8797.156671] Lustre: 344372:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8797.202740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8797.205023] Lustre: Skipped 1 previous similar message [ 8797.243944] Lustre: lustre-MDT0001: new disk, initializing [ 8797.285735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8797.304533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8797.307666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8798.719084] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8802.093769] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8803.382563] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8805.884644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8805.910720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8806.014915] Lustre: lustre-OST0000: new disk, initializing [ 8806.017376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8806.051684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.935306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8807.938612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8807.973118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8808.143553] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8811.416799] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8813.022464] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8813.099393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8814.582642] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8814.675813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8818.148517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8818.148825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8818.152440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8818.156682] Lustre: Skipped 1 previous similar message [ 8821.487062] LustreError: 347310:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8821.524031] Lustre: server umount lustre-OST0000 complete [ 8823.264610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.264644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8823.266555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8823.266560] Lustre: Skipped 1 previous similar message [ 8823.269919] Lustre: Skipped 1 previous similar message [ 8823.278432] LustreError: Skipped 1 previous similar message [ 8828.384182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8828.386391] Lustre: Skipped 2 previous similar messages [ 8828.989570] LustreError: 347511:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8828.991754] LustreError: 347511:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8829.019067] Lustre: server umount lustre-MDT0000 complete [ 8830.481748] LustreError: 345152:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081843 with bad export cookie 1036381797970932627 [ 8830.484172] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8830.487915] LustreError: 345152:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8830.559071] LustreError: 347712:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8830.561788] LustreError: 347712:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8830.617818] Lustre: server umount lustre-MDT0001 complete [ 8833.500823] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 06:37:25 (1769081845) [ 8835.419453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8837.607279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8845.787232] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8848.821048] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8852.331908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8854.578948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8856.644147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8859.059450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8861.670170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8861.705739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.822894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8861.835713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8861.881387] Lustre: lustre-MDT0000: new disk, initializing [ 8861.924755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.934943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8863.451621] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8867.917438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.942517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8867.974178] Lustre: 351481:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8867.999616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8868.002018] Lustre: Skipped 1 previous similar message [ 8868.040213] Lustre: lustre-MDT0001: new disk, initializing [ 8868.075731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8868.088724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8868.093293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8869.469020] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8872.723056] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8874.030789] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8876.519621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8876.543807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8876.636742] Lustre: lustre-OST0000: new disk, initializing [ 8876.638933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8876.669241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8878.604350] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8881.775123] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8882.679480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8882.682894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8882.728279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8883.178508] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8883.247732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8884.580233] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8884.653033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8887.777267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8887.777697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.780045] LustreError: Skipped 1 previous similar message [ 8887.784810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8887.788542] Lustre: Skipped 1 previous similar message [ 8891.374095] LustreError: 354418:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8891.376280] LustreError: 354418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8891.415473] Lustre: server umount lustre-OST0000 complete [ 8892.896710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8892.897223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8892.901435] Lustre: Skipped 2 previous similar messages [ 8898.750148] LustreError: 354619:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8898.752253] LustreError: 354619:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8898.779353] Lustre: server umount lustre-MDT0000 complete [ 8900.009160] LustreError: 351472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769081913 with bad export cookie 1036381797970933495 [ 8900.012325] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8900.014295] LustreError: 351472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8900.128781] Lustre: server umount lustre-MDT0001 complete [ 8902.927438] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:38:35 (1769081915) [ 8905.389281] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:38:37 (1769081917) [ 8908.473628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8908.679665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.019505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8913.081484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8914.783368] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8916.701141] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8917.892839] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8920.404232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8922.595893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8924.448296] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8938.363606] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8938.907779] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8939.479740] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8940.091269] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8940.817770] Lustre: Failing over lustre-MDT0000 [ 8940.970365] LustreError: 358296:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8940.972838] LustreError: 358296:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8941.002504] Lustre: server umount lustre-MDT0000 complete [ 8943.073619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8943.074976] LustreError: 355830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.078836] Lustre: Skipped 2 previous similar messages [ 8943.087886] LustreError: 355830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8945.122489] LustreError: 357450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.192035] LustreError: 355834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.198116] LustreError: 355834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8950.231102] LustreError: 355830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8954.032626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.073235] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8954.163165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8954.165392] Lustre: Skipped 2 previous similar messages [ 8954.184594] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8955.350872] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8955.496206] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8959.461028] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8959.466505] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8959.483355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8960.646716] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8961.188053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8962.997967] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8963.633973] Lustre: Failing over lustre-MDT0000 [ 8963.822746] LustreError: 359374:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8963.825588] LustreError: 359374:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8963.850471] Lustre: server umount lustre-MDT0000 complete [ 8964.576391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8964.576451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8964.578833] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8964.578840] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8964.590455] Lustre: Skipped 2 previous similar messages [ 8974.817133] LustreError: 355853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.823500] LustreError: 355853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 8976.710834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8976.753920] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8976.863561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8978.154805] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8980.952608] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8981.992928] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8981.995903] Lustre: Skipped 2 previous similar messages [ 8982.010757] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8982.029034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8983.320672] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8983.965313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8987.105044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8987.105360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8987.109424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8987.109433] Lustre: Skipped 3 previous similar messages [ 8987.119690] Lustre: Skipped 2 previous similar messages [ 8992.535469] Lustre: server umount lustre-MDT0000 complete [ 8993.804261] LustreError: 355814:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082007 with bad export cookie 1036381797970936176 [ 8993.806814] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8993.808687] LustreError: 355814:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8993.924725] Lustre: server umount lustre-MDT0001 complete [ 9005.498222] LustreError: 360808:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9005.500208] LustreError: 360808:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9005.560147] Lustre: server umount lustre-OST0000 complete [ 9011.247255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.452788] LustreError: 361635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9011.464510] LustreError: 361635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9012.930632] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9016.087344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9017.755371] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9019.697453] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9020.855902] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9023.242615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9023.375573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9023.378389] Lustre: Skipped 3 previous similar messages [ 9025.398370] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9027.296440] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9035.685718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9040.866710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9040.868658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9040.869776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9040.869783] Lustre: Skipped 5 previous similar messages [ 9040.874020] Lustre: Skipped 1 previous similar message [ 9045.404753] Lustre: server umount lustre-OST0000 complete [ 9061.343102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9061.405994] Lustre: server umount lustre-MDT0000 complete [ 9062.622128] LustreError: 361660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082075 with bad export cookie 1036381797970936883 [ 9062.624209] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.626737] LustreError: 361660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9062.744873] Lustre: server umount lustre-MDT0001 complete [ 9066.123926] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9067.129991] Key type lgssc unregistered [ 9067.263464] LNet: 364836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9067.266496] LNetError: 364836:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9067.274333] LNet: Removed LNI 192.168.203.155@tcp [ 9067.591375] Key type .llcrypt unregistered [ 9067.593092] Key type ._llcrypt unregistered [ 9077.956365] Key type ._llcrypt registered [ 9077.957911] Key type .llcrypt registered [ 9078.005728] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9078.432786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9078.444142] alg: No test for adler32 (adler32-zlib) [ 9079.305431] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9079.398313] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9080.983198] Key type lgssc registered [ 9081.409292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9084.293790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9084.300193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.457687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9086.931250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9090.054482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9090.186854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9091.668174] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9093.477726] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9094.727968] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9097.100754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9097.224707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.226196] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9101.112122] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9110.565652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9115.617436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.618407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9115.621703] Lustre: Skipped 2 previous similar messages [ 9115.625235] Lustre: Skipped 2 previous similar messages [ 9120.737017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9120.738916] Lustre: Skipped 2 previous similar messages [ 9121.469444] LustreError: 368576:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9121.514169] Lustre: server umount lustre-MDT0000 complete [ 9122.790067] LustreError: 367029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082136 with bad export cookie 16783778331720324697 [ 9122.792905] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9122.794416] LustreError: 367029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9122.867434] LustreError: 368779:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9122.869192] LustreError: 368779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9122.915233] Lustre: server umount lustre-MDT0001 complete [ 9134.322881] LustreError: 368979:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9134.324910] LustreError: 368979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9134.386813] Lustre: server umount lustre-OST0000 complete [ 9138.992655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.132208] LustreError: 369709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9139.138676] LustreError: 369709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9139.155082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9140.421493] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9143.531377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9143.656486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9145.030966] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9146.900720] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9148.291756] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9150.670641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9153.066117] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9154.928202] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9164.134601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9174.498995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.501182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.503400] Lustre: Skipped 2 previous similar messages [ 9174.505724] Lustre: Skipped 1 previous similar message [ 9174.508718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9178.366785] LustreError: 372076:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9178.368791] LustreError: 372076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9178.399196] Lustre: server umount lustre-MDT0000 complete [ 9179.617575] LustreError: 371013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.623156] LustreError: 371013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9179.654059] LustreError: 369688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082192 with bad export cookie 16783778331720325796 [ 9179.656969] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9179.657510] LustreError: 369688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9179.768364] Lustre: server umount lustre-MDT0001 complete [ 9191.282257] LustreError: 372481:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9191.285140] LustreError: 372481:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9191.336516] Lustre: server umount lustre-OST0000 complete [ 9196.610211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9196.758041] LustreError: 373308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9196.781439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9196.784366] Lustre: Skipped 1 previous similar message [ 9198.052748] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9201.033362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9202.590584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9204.487823] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9205.713219] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9208.344046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9210.681749] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9212.729712] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9221.796648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9227.232802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9227.235397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9227.235870] LustreError: Skipped 1 previous similar message [ 9227.240547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9227.244592] Lustre: Skipped 2 previous similar messages [ 9230.445563] LustreError: 375578:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9230.447953] LustreError: 375578:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9230.482447] Lustre: server umount lustre-OST0000 complete [ 9231.839868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9231.843373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9231.847870] Lustre: Skipped 1 previous similar message [ 9231.849967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.592058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.593913] Lustre: Skipped 5 previous similar messages [ 9246.175153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9246.239423] Lustre: server umount lustre-MDT0000 complete [ 9246.688399] LustreError: 373308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9246.696030] LustreError: 373308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9247.636265] LustreError: 373289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082260 with bad export cookie 16783778331720326804 [ 9247.639316] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9247.642454] LustreError: 373289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9247.710042] LustreError: 375981:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9247.712699] LustreError: 375981:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9247.782766] Lustre: server umount lustre-MDT0001 complete [ 9251.819576] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9252.972860] Key type lgssc unregistered [ 9253.106302] LNet: 376513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9253.109173] LNetError: 376513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9253.122436] LNet: Removed LNI 192.168.203.155@tcp [ 9253.506373] Key type .llcrypt unregistered [ 9253.507898] Key type ._llcrypt unregistered [ 9263.566124] Key type ._llcrypt registered [ 9263.567471] Key type .llcrypt registered [ 9263.607137] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9263.936024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9263.984641] alg: No test for adler32 (adler32-zlib) [ 9264.841980] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9264.923844] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9266.503151] Key type lgssc registered [ 9266.865967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9269.430297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9269.436489] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.439696] LustreError: 377909:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9269.447982] LustreError: 377909:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9269.450167] LustreError: 377909:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9269.453162] LustreError: 377909:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9269.455855] LustreError: 377909:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9272.043784] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9272.046831] LustreError: 378307:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9272.055764] LustreError: 378307:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9272.058046] LustreError: 378307:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9272.060339] LustreError: 378307:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9272.063054] LustreError: 378307:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9278.430548] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9279.402783] Key type lgssc unregistered [ 9279.522302] LNet: 379292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9279.524773] LNetError: 379292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9279.536264] LNet: Removed LNI 192.168.203.155@tcp [ 9279.829437] Key type .llcrypt unregistered [ 9279.830594] Key type ._llcrypt unregistered [ 9286.094402] Key type ._llcrypt registered [ 9286.095968] Key type .llcrypt registered [ 9286.146904] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9292.231394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9292.638638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9292.650573] alg: No test for adler32 (adler32-zlib) [ 9293.497260] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9293.577433] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9295.159146] Key type lgssc registered [ 9295.523884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9298.467638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9300.472867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9302.651334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9304.798795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9307.133950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9307.154377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9307.159096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.244053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9308.256117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9308.290130] Lustre: lustre-MDT0000: new disk, initializing [ 9308.313341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9308.320644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9309.695021] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9314.111740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9314.139519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9314.172965] Lustre: 382396:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9314.199495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9314.202180] Lustre: Skipped 1 previous similar message [ 9314.239485] Lustre: lustre-MDT0001: new disk, initializing [ 9314.276181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9314.289635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9314.293191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9315.677835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9318.982310] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9320.108263] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9322.391796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9322.425672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9322.525473] Lustre: lustre-OST0000: new disk, initializing [ 9322.527736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9322.558574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9324.027481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9324.031563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9324.072139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9324.592915] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9327.880644] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9329.264974] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9329.342125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9330.750569] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9330.832563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9333.549148] LustreError: 385337:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9333.587405] Lustre: server umount lustre-OST0000 complete [ 9339.361365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.362065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9339.366879] Lustre: Skipped 1 previous similar message [ 9344.480122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9344.482294] Lustre: Skipped 2 previous similar messages [ 9345.506919] LustreError: 385537:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9345.509019] LustreError: 385537:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9345.534387] Lustre: server umount lustre-MDT0000 complete [ 9346.893253] LustreError: 383178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082360 with bad export cookie 517132081032288051 [ 9346.894996] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9346.899519] LustreError: 383178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9357.289389] LustreError: 385741:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9357.294989] LustreError: 385741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9357.359132] Lustre: server umount lustre-MDT0001 complete [ 9360.726402] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9361.311919] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:46:13 (1769082373) [ 9363.972514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9364.282322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.966994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9369.147678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9369.282939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9370.658883] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9372.568189] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9373.815282] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9376.114572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9378.217741] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9379.994288] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9382.398718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9382.429389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9382.508482] Lustre: lustre-OST0001: new disk, initializing [ 9382.511203] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9382.543606] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9382.545882] Lustre: Skipped 1 previous similar message [ 9384.586439] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9387.864450] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9389.077427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9389.083389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9389.118797] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9394.145612] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.145995] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 9394.146348] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.148400] LustreError: Skipped 1 previous similar message [ 9394.154318] Lustre: Skipped 1 previous similar message [ 9395.882965] LustreError: 389654:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9395.884933] LustreError: 389654:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9395.903963] Lustre: server umount lustre-OST0001 complete [ 9399.265930] LustreError: 388058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9399.271180] LustreError: 388058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9404.392902] LustreError: 388056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9404.398016] LustreError: 388056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9409.506500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.506728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9409.507335] LustreError: 389658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9409.508280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9409.511814] Lustre: Skipped 2 previous similar messages [ 9414.625425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9414.627371] LustreError: 388058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.627764] Lustre: Skipped 2 previous similar messages [ 9414.633578] LustreError: 388058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9419.746819] LustreError: 388057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9419.752326] LustreError: 388057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9422.303131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9422.329087] LustreError: 389880:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9422.334438] LustreError: 389880:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9422.374234] Lustre: server umount lustre-OST0000 complete [ 9424.864738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.865584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.868639] Lustre: Skipped 1 previous similar message [ 9424.870608] Lustre: Skipped 2 previous similar messages [ 9429.842980] Lustre: server umount lustre-MDT0000 complete [ 9429.984038] LustreError: 389656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9429.989602] LustreError: 389656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9431.075440] LustreError: 386441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082444 with bad export cookie 517132081032288919 [ 9431.078245] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9431.079029] LustreError: 386441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9445.343121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9445.378438] LustreError: 390284:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9445.380869] LustreError: 390284:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9445.437567] Lustre: server umount lustre-MDT0001 complete [ 9449.131584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9450.165333] Key type lgssc unregistered [ 9450.291328] LNet: 390816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9450.293950] LNetError: 390816:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9450.305411] LNet: Removed LNI 192.168.203.155@tcp [ 9450.601352] Key type .llcrypt unregistered [ 9450.602503] Key type ._llcrypt unregistered [ 9461.909220] Key type ._llcrypt registered [ 9461.910836] Key type .llcrypt registered [ 9461.948116] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9468.090701] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9468.549576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9468.562372] alg: No test for adler32 (adler32-zlib) [ 9469.413465] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9469.501672] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9471.079127] Key type lgssc registered [ 9471.438700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9474.297134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9476.402476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9478.454389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9480.553751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9482.854320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9482.883499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9482.896785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.993313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9484.006341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9484.041877] Lustre: lustre-MDT0000: new disk, initializing [ 9484.066625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9484.073770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9485.440874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9489.648340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9489.670978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9489.703457] Lustre: 394736:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9489.725439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9489.728115] Lustre: Skipped 1 previous similar message [ 9489.762427] Lustre: lustre-MDT0001: new disk, initializing [ 9489.794439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9489.806816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9489.810544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9491.082979] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9494.218264] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9495.369125] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9497.773417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9497.807884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9497.920769] Lustre: lustre-OST0000: new disk, initializing [ 9497.923469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9497.967422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9498.810704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9498.814711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9498.860198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9500.006545] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9503.264285] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9504.831774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9504.916753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9506.395914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9506.486532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9509.344131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9509.347458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9509.350631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9509.351757] Lustre: Skipped 1 previous similar message [ 9513.260698] LustreError: 397677:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9513.299518] Lustre: server umount lustre-OST0000 complete [ 9515.487884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9515.491906] LustreError: Skipped 1 previous similar message [ 9515.493941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9515.500247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9515.502327] Lustre: Skipped 1 previous similar message [ 9519.586047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.586271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9519.592820] Lustre: Skipped 1 previous similar message [ 9520.703070] LustreError: 397879:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9520.705110] LustreError: 397879:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9520.735416] Lustre: server umount lustre-MDT0000 complete [ 9521.927094] LustreError: 394727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082535 with bad export cookie 11412482665039948610 [ 9521.930227] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9521.931102] LustreError: 394727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9521.988081] LustreError: 398082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9521.990076] LustreError: 398082:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9522.036349] Lustre: server umount lustre-MDT0001 complete [ 9524.893285] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 06:48:57 (1769082537) [ 9527.891766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9530.642835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9530.790921] Lustre: lustre-MDT0000: new disk, initializing [ 9530.825774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.090540] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9535.154757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9535.316355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9536.607272] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9538.377995] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9539.624707] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9540.328253] LustreError: 399376:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 9540.330285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9540.330752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9540.331544] LustreError: 399376:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 9540.335496] Lustre: Skipped 1 previous similar message [ 9540.337301] LustreError: 399376:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 9545.440231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9545.442904] Lustre: Skipped 2 previous similar messages [ 9546.502283] LustreError: 400689:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9546.509917] LustreError: 400689:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9546.591232] Lustre: server umount lustre-MDT0000 complete [ 9547.942336] LustreError: 399327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082561 with bad export cookie 11412482665039949478 [ 9547.945666] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.947138] LustreError: 399327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9548.057177] Lustre: server umount lustre-MDT0001 complete [ 9551.255614] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:49:23 (1769082563) [ 9553.571954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9553.737795] LustreError: 401614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.746245] LustreError: 401614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9553.764912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.125427] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9557.904687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9559.361170] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9561.216687] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9562.403704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9564.881077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9566.945708] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9568.889234] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9583.259533] Lustre: Permanently deactivating lustre-OST0000 [ 9583.264504] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9593.324866] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9594.090813] LustreError: 404468:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9594.094131] LustreError: 404468:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 9594.096039] LustreError: 404468:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9594.680034] LustreError: 404517:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 9594.682833] LustreError: 404517:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9601.325458] LustreError: 404615:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9601.327336] LustreError: 404615:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9601.356065] Lustre: server umount lustre-OST0000 complete [ 9602.217630] LustreError: 404767:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 9602.219806] LustreError: 404767:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9604.063715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9604.066528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.071408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.375191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9617.412405] LustreError: 404864:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9617.415117] LustreError: 404864:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9617.436092] Lustre: server umount lustre-MDT0000 complete [ 9618.743207] LustreError: 403979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082632 with bad export cookie 11412482665039950010 [ 9618.746621] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9618.748574] LustreError: 403979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9629.206884] Lustre: server umount lustre-MDT0001 complete [ 9632.242322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.592019] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9634.535863] LustreError: 405908:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9636.777449] Lustre: 406246:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [ 9637.537451] LustreError: 406345:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9637.540101] LustreError: 406345:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9637.565722] Lustre: server umount MGS complete [ 9641.391891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.547026] LustreError: 406972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9641.552892] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9641.558746] Lustre: Skipped 1 previous similar message [ 9641.580692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9641.583195] Lustre: Skipped 2 previous similar messages [ 9642.866633] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9645.683343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9645.781538] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9647.180093] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9649.011917] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9650.206409] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9650.852089] Lustre: Permanently reactivating lustre-OST0000 [ 9650.856433] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9650.858473] Lustre: Skipped 2 previous similar messages [ 9672.671159] LustreError: 408390:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9675.013929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9675.108251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9675.110713] Lustre: Skipped 1 previous similar message [ 9676.914843] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9678.696879] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9692.639870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9692.642751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9692.647450] Lustre: Skipped 1 previous similar message [ 9692.650157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9692.653103] Lustre: Skipped 6 previous similar messages [ 9696.301967] LustreError: 410208:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9696.304134] LustreError: 410208:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9696.341178] Lustre: server umount lustre-OST0000 complete [ 9703.637218] Lustre: server umount lustre-MDT0000 complete [ 9703.904906] LustreError: 406972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.909883] LustreError: 406972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9704.757286] LustreError: 406951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082718 with bad export cookie 11412482665039950955 [ 9704.760184] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9704.761203] LustreError: 406951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9704.855178] Lustre: server umount lustre-MDT0001 complete [ 9708.002824] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9708.942848] Key type lgssc unregistered [ 9709.067357] LNet: 411143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9709.070514] LNetError: 411143:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9709.079313] LNet: Removed LNI 192.168.203.155@tcp [ 9709.335444] Key type .llcrypt unregistered [ 9709.336648] Key type ._llcrypt unregistered [ 9714.812747] Key type ._llcrypt registered [ 9714.813816] Key type .llcrypt registered [ 9714.847425] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9720.882281] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9721.211740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9721.250690] alg: No test for adler32 (adler32-zlib) [ 9722.100751] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9722.184180] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9723.759290] Key type lgssc registered [ 9724.149236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9727.195386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9729.388065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9731.498770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9733.705521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9735.903693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9735.922385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9735.927368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.008969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9737.019438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9737.054891] Lustre: lustre-MDT0000: new disk, initializing [ 9737.077094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.082978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9738.270138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9742.230324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9742.253192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9742.281826] Lustre: 414245:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9742.303687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9742.305511] Lustre: Skipped 1 previous similar message [ 9742.336786] Lustre: lustre-MDT0001: new disk, initializing [ 9742.365058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9742.375401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9742.378699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9743.643956] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9746.583057] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9747.711719] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9749.852859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9749.880372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9749.963727] Lustre: lustre-OST0000: new disk, initializing [ 9749.965305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9749.993695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9750.776497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9750.781116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9750.810094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9751.849591] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9754.985349] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9756.350788] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9756.433525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9757.797266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9757.879169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9760.558077] LustreError: 417183:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9760.598025] Lustre: server umount lustre-OST0000 complete [ 9762.783839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9762.786269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.790330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.369356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.369930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.488426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9771.490423] Lustre: Skipped 2 previous similar messages [ 9772.516326] LustreError: 417384:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9772.518586] LustreError: 417384:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9772.545803] Lustre: server umount lustre-MDT0000 complete [ 9773.842454] LustreError: 414235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082787 with bad export cookie 13837638944400706413 [ 9773.845717] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9773.846919] LustreError: 414235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9784.290989] LustreError: 417586:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9784.292770] LustreError: 417586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9784.334754] Lustre: server umount lustre-MDT0001 complete [ 9787.073990] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9787.649697] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:53:19 (1769082799) [ 9789.960746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9790.131932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.405114] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9794.279470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9794.423410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9795.869570] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9797.720212] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9798.971942] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9801.332701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9803.412917] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9805.244699] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9805.923246] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9816.605039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9820.128588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9820.132894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.139819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9821.152708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9824.173756] LustreError: 420761:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9824.175754] LustreError: 420761:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9824.213934] Lustre: server umount lustre-OST0000 complete [ 9826.273086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.278043] Lustre: Skipped 1 previous similar message [ 9826.280398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.282733] Lustre: Skipped 3 previous similar messages [ 9831.613730] LustreError: 420964:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9831.615849] LustreError: 420964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9831.643561] Lustre: server umount lustre-MDT0000 complete [ 9832.934256] LustreError: 418284:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082846 with bad export cookie 13837638944400707274 [ 9832.937610] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9832.938067] LustreError: 418284:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9833.057916] Lustre: server umount lustre-MDT0001 complete [ 9836.703249] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9837.667210] Key type lgssc unregistered [ 9837.795419] LNet: 421696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9837.799208] LNetError: 421696:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9837.807306] LNet: Removed LNI 192.168.203.155@tcp [ 9838.084673] Key type .llcrypt unregistered [ 9838.085728] Key type ._llcrypt unregistered [ 9845.685155] Key type ._llcrypt registered [ 9845.686478] Key type .llcrypt registered [ 9845.723832] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9845.826241] Key type .llcrypt unregistered [ 9845.828100] Key type ._llcrypt unregistered [ 9852.809627] Key type ._llcrypt registered [ 9852.811017] Key type .llcrypt registered [ 9852.851374] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9853.130522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9853.136206] alg: No test for adler32 (adler32-zlib) [ 9853.971720] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9854.044247] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9855.615141] Key type lgssc registered [ 9855.934742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9858.234469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9858.240184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.375904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.553695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9863.154679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9863.253457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9864.907461] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9866.522430] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9868.257430] LustreError: 423863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9868.261672] LustreError: 423863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9868.755731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9868.867542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9869.933577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9870.197431] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9873.888056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9873.890114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9873.893585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9875.425237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9875.425381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9875.426855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9879.276108] LustreError: 425898:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9879.309838] Lustre: server umount lustre-OST0000 complete [ 9884.127992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9884.129384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.130770] LustreError: Skipped 1 previous similar message [ 9884.137112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.139113] Lustre: Skipped 1 previous similar message [ 9886.717633] LustreError: 426101:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9886.720028] LustreError: 426101:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9886.752598] Lustre: server umount lustre-MDT0000 complete [ 9887.992201] LustreError: 423825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769082901 with bad export cookie 17357357777998808932 [ 9887.995989] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9887.996967] LustreError: 423825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9888.070780] LustreError: 426302:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9888.073244] LustreError: 426302:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9888.128866] Lustre: server umount lustre-MDT0001 complete [ 9892.147265] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9893.326091] Key type lgssc unregistered [ 9893.460684] LNet: 426833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9893.469989] LNetError: 426833:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9893.491445] LNet: Removed LNI 192.168.203.155@tcp [ 9893.834524] Key type .llcrypt unregistered [ 9893.836043] Key type ._llcrypt unregistered [ 9903.121642] Key type ._llcrypt registered [ 9903.122795] Key type .llcrypt registered [ 9903.158631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9903.577328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9903.583223] alg: No test for adler32 (adler32-zlib) [ 9904.434022] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [ 9904.515710] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9906.095182] Key type lgssc registered [ 9906.480836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9909.102732] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9909.108320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9924.575274] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9929.695198] LustreError: 428180:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 9956.319420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9958.105245] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9960.031514] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9962.138705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9962.322589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9962.343254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9963.526927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9966.508444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9966.621045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9967.829926] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9969.499833] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9970.604655] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9977.823944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9977.826804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9977.831574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9980.898121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9980.898737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.956221] LustreError: 430461:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 9983.000513] Lustre: server umount lustre-OST0000 complete [ 9986.016867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9986.017679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9986.021151] Lustre: Skipped 1 previous similar message [ 9986.025841] Lustre: Skipped 2 previous similar messages [ 9990.527905] LustreError: 430663:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9990.530399] LustreError: 430663:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9990.566320] Lustre: server umount lustre-MDT0000 complete [ 9991.135874] LustreError: 429107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.140993] LustreError: 429107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9991.857305] LustreError: 428187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083005 with bad export cookie 15220857288428799544 [ 9991.860409] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9991.861692] LustreError: 428187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9991.928967] LustreError: 430865:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9991.931498] LustreError: 430865:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9991.988241] Lustre: server umount lustre-MDT0001 complete [ 9996.560380] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9997.927498] Key type lgssc unregistered [ 9998.082334] LNet: 431396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9998.085026] LNetError: 431396:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9998.096435] LNet: Removed LNI 192.168.203.155@tcp [ 9998.422336] Key type .llcrypt unregistered [ 9998.423692] Key type ._llcrypt unregistered [10011.190996] Key type ._llcrypt registered [10011.192545] Key type .llcrypt registered [10011.238163] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10011.595110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10011.621087] alg: No test for adler32 (adler32-zlib) [10012.476423] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [10012.560353] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10014.151188] Key type lgssc registered [10014.501605] Lustre: Echo OBD driver; http://www.lustre.org/ [10017.123081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10017.130832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10018.313759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.637967] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10022.752121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10022.891474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10024.277728] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10026.156448] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10027.360322] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10029.632695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10029.751180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10031.556234] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10033.303938] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10043.045405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10048.482180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10048.482641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10048.483266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10048.485907] Lustre: Skipped 2 previous similar messages [10048.990657] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [10048.993579] Lustre: Skipped 2 previous similar messages [10050.817409] LustreError: 435038:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10050.855457] Lustre: server umount lustre-MDT0000 complete [10051.649542] LustreError: 434388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.55@tcp arrived at 1769083064 with bad export cookie 1087715270247334609 [10051.653243] LustreError: 434388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10052.283966] LustreError: 434388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083065 with bad export cookie 1087715270247333958 [10052.287821] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10052.288165] LustreError: 434388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10053.600264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10053.600507] LustreError: 432771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10053.600627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.601481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10053.615110] LustreError: 432771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10057.631771] LustreError: 432771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.632042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10057.636312] LustreError: 432771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10058.351084] LustreError: 435241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10058.353426] LustreError: 435241:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10058.388452] Lustre: server umount lustre-OST0000 complete [10060.647568] LustreError: 435595:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10060.650493] LustreError: 435595:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10060.712619] Lustre: server umount lustre-MDT0001 complete [10064.348538] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10065.347478] Key type lgssc unregistered [10065.465367] LNet: 436125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10065.467808] LNetError: 436125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10065.475350] LNet: Removed LNI 192.168.203.155@tcp [10065.762187] Key type .llcrypt unregistered [10065.763531] Key type ._llcrypt unregistered [10074.895450] Key type ._llcrypt registered [10074.896389] Key type .llcrypt registered [10074.928453] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10075.334686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10075.340794] alg: No test for adler32 (adler32-zlib) [10076.187266] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [10076.264880] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10077.839154] Key type lgssc registered [10078.176752] Lustre: Echo OBD driver; http://www.lustre.org/ [10080.662447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10080.669829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.819518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10083.000929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10085.635392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10085.741746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10086.945861] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10088.686819] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10089.844976] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10092.063772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10092.193426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10094.019232] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10095.731941] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10107.557847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10112.995465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10112.999874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10113.001703] Lustre: Skipped 1 previous similar message [10116.577085] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [10118.113195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10118.115558] Lustre: Skipped 1 previous similar message [10118.399495] LustreError: 439766:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10118.427401] Lustre: server umount lustre-MDT0001 complete [10123.233304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10123.233317] LustreError: 438913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.234843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10123.234849] Lustre: Skipped 1 previous similar message [10123.235431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10123.252646] LustreError: 438913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10126.253367] LustreError: 439973:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10126.255377] LustreError: 439973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10126.292195] Lustre: server umount lustre-OST0000 complete [10127.582001] LustreError: 440174:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10127.584296] LustreError: 440174:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10127.644694] Lustre: server umount lustre-MDT0000 complete [10132.227394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10133.241940] Key type lgssc unregistered [10133.363524] LNet: 440858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10133.366191] LNetError: 440858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10133.374268] LNet: Removed LNI 192.168.203.155@tcp [10133.646522] Key type .llcrypt unregistered [10133.647669] Key type ._llcrypt unregistered [10141.704175] Key type ._llcrypt registered [10141.705285] Key type .llcrypt registered [10141.741962] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10147.716894] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10148.111193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10148.170410] alg: No test for adler32 (adler32-zlib) [10149.026341] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [10149.131401] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10150.719174] Key type lgssc registered [10151.096962] Lustre: Echo OBD driver; http://www.lustre.org/ [10154.151432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10156.249510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10158.268174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10160.337061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10163.598725] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10166.464810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10168.738055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10168.763704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10168.769404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10169.867333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10169.878836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10169.917479] Lustre: lustre-MDT0000: new disk, initializing [10169.946393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10169.954231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10171.258940] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10174.747608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10176.955163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10176.977053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10177.083092] Lustre: lustre-OST0000: new disk, initializing [10177.084786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10177.086239] Lustre: Skipped 1 previous similar message [10177.110388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10178.921681] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10181.887433] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10186.745301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10186.748681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10186.756299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10190.380131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10192.637180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10192.660191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10192.698973] Lustre: 444803:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10192.728606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10192.768419] Lustre: lustre-MDT0001: new disk, initializing [10192.813837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10192.827229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10192.830058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10194.149266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10197.754676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10197.823356] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10199.220949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10202.357947] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [10206.690356] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.690415] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10206.695558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10206.699045] Lustre: Skipped 1 previous similar message [10208.734929] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [10210.044130] LustreError: 448249:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10210.067072] Lustre: server umount lustre-MDT0001 complete [10216.929532] LustreError: 444839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.935103] LustreError: 444839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10222.050992] LustreError: 444810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.055974] LustreError: 444810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10222.559902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10222.563657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10222.568647] Lustre: Skipped 1 previous similar message [10222.571190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10227.168423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10227.168948] LustreError: 444814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.463559] LustreError: 448522:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [10228.466718] LustreError: 448522:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10228.544426] Lustre: server umount lustre-OST0000 complete [10237.022855] LustreError: 448723:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [10237.026239] LustreError: 448723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10237.159275] Lustre: server umount lustre-MDT0000 complete [10248.000266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10250.653543] Key type lgssc unregistered [10250.912662] LNet: 449419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10250.921175] LNetError: 449419:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10250.940355] LNet: Removed LNI 192.168.203.155@tcp [10251.675888] Key type .llcrypt unregistered [10251.677578] Key type ._llcrypt unregistered [10286.452869] Key type ._llcrypt registered [10286.458903] Key type .llcrypt registered [10286.544294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10286.702605] Key type .llcrypt unregistered [10286.703915] Key type ._llcrypt unregistered [10295.754572] Key type ._llcrypt registered [10295.755889] Key type .llcrypt registered [10295.814576] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10296.183652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10296.197944] alg: No test for adler32 (adler32-zlib) [10297.137449] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [10297.263512] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10298.871155] Key type lgssc registered [10299.384696] Lustre: Echo OBD driver; http://www.lustre.org/ [10303.214505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10306.381283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10309.267058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10312.679180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10312.725678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10312.735028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.859986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10313.874378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10313.939165] Lustre: lustre-MDT0000: new disk, initializing [10314.027197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10314.037908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10316.095076] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10322.100371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10322.137134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10322.196702] Lustre: 452748:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10322.230204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10322.232631] Lustre: Skipped 1 previous similar message [10322.277416] Lustre: lustre-MDT0001: new disk, initializing [10322.343860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10322.379703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10322.384449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10324.627442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10329.253656] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10331.470341] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10335.248289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10335.283097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10335.422732] Lustre: lustre-OST0000: new disk, initializing [10335.426210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10335.487321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10338.973659] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10343.798665] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10344.466662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10344.472119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10344.538648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10353.120440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10353.127320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.149690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.658370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.658630] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.662543] Lustre: Skipped 1 previous similar message [10354.675628] Lustre: Skipped 1 previous similar message [10359.066625] LustreError: 455186:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10359.109342] Lustre: server umount lustre-MDT0000 complete [10361.047904] LustreError: 452741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083374 with bad export cookie 18064490891799866445 [10361.050298] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10361.053699] LustreError: 452741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10361.132576] LustreError: 455387:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10361.135065] LustreError: 455387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10361.234890] Lustre: server umount lustre-MDT0001 complete [10373.367236] LustreError: 455590:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10373.370278] LustreError: 455590:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10373.442327] Lustre: server umount lustre-OST0000 complete [10378.793348] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:03:10 (1769083390) [10379.218276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10382.676875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.887863] LustreError: 456418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.900017] LustreError: 456418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10382.936157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10384.695182] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10387.936565] LustreError: 456418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.687932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10388.897274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10391.065458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10394.307607] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10396.290458] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10400.063578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10400.116185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10400.208325] Lustre: Found index 0 for lustre-OST0000, updating log [10400.319398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10403.542541] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10406.930347] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10408.496090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10415.522372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10415.531267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10415.539491] Lustre: Skipped 2 previous similar messages [10420.593256] LustreError: 458674:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10420.595989] LustreError: 458674:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10420.662744] Lustre: server umount lustre-OST0000 complete [10422.054485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10424.800219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10424.807072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.816164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10424.819493] Lustre: Skipped 2 previous similar messages [10428.996451] LustreError: 458929:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10429.001392] LustreError: 458929:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10429.033634] Lustre: server umount lustre-MDT0000 complete [10430.899183] LustreError: 456397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083444 with bad export cookie 18064490891799867824 [10430.903698] LustreError: 456397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10430.906884] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10431.074263] Lustre: server umount lustre-MDT0001 complete [10435.116398] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:04:07 (1769083447) [10435.558653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10435.955721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10436.364110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10436.777505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10437.186905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10437.599311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10438.013149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10438.429189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10438.845913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10439.273374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10439.718490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10440.191716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10440.628630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10441.044720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10441.393587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10441.783919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10442.217611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10442.628490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10443.045345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10443.487429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10443.926676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10444.414624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10444.930509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10445.399166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10445.865606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10446.319622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10446.754221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10447.144779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10447.572196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10448.015534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10448.450636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10448.866582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10449.286201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10449.698087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10450.096938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10450.453218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10450.825274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10451.238923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10451.658660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10452.064516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10452.462644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10452.819976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10453.179776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10453.581158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10453.950188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10454.437991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10454.863068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10455.248207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10455.646404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10456.062628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10456.445340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10458.080069] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [10459.692973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10459.868227] LustreError: 462516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.899180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10461.542339] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10465.251923] LustreError: 462516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10465.920118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10468.251810] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10471.144087] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10472.874988] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10476.272734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10476.346600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10476.473789] Lustre: Found index 0 for lustre-OST0000, updating log [10479.619149] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10482.159984] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10483.171204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.180244] Lustre: Skipped 1 previous similar message [10483.182597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10483.183135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10483.194915] Lustre: Skipped 2 previous similar messages [10489.135122] LustreError: 464776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10489.140258] LustreError: 464776:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10489.205552] Lustre: server umount lustre-OST0000 complete [10491.810111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.812354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10491.821595] Lustre: Skipped 1 previous similar message [10491.827603] Lustre: Skipped 1 previous similar message [10505.183129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10505.223918] LustreError: 464986:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10505.227123] LustreError: 464986:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10505.259881] Lustre: server umount lustre-MDT0000 complete [10506.209060] LustreError: 462515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.218519] LustreError: 462515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10507.106170] LustreError: 463236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083520 with bad export cookie 18064490891799868895 [10507.107297] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10507.113878] LustreError: 463236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10507.273275] Lustre: server umount lustre-MDT0001 complete [10508.499527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10515.772830] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10519.753954] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10523.680629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10526.361625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10526.687100] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [10529.097703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10532.106720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10535.183535] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:05:47 (1769083547) [10542.511781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10543.759350] Key type lgssc unregistered [10543.917451] LNet: 468751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10543.920766] LNetError: 468751:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10667.174303] LNet: Removed LNI 192.168.203.155@tcp [10667.788191] Key type .llcrypt unregistered [10667.790242] Key type ._llcrypt unregistered [10678.782640] Key type ._llcrypt registered [10678.784478] Key type .llcrypt registered [10678.856163] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10679.612341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10679.675443] alg: No test for adler32 (adler32-zlib) [10680.710675] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [10680.928803] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10682.567494] Key type lgssc registered [10683.279468] Lustre: Echo OBD driver; http://www.lustre.org/ [10686.068658] LNet: Added LNI 192.168.203.155@tcp100 [8/256/0/180] [10686.619350] LNet: Added LNI 192.168.203.155@tcp101 [8/256/0/180] [10687.274681] LNet: Added LNI 192.168.203.155@tcp102 [8/256/0/180] [10687.936324] LNet: Added LNI 192.168.203.155@tcp103 [8/256/0/180] [10688.502640] LNet: Added LNI 192.168.203.155@tcp104 [8/256/0/180] [10689.093154] LNet: Added LNI 192.168.203.155@tcp105 [8/256/0/180] [10689.636979] LNet: Added LNI 192.168.203.155@tcp106 [8/256/0/180] [10690.179731] LNet: Added LNI 192.168.203.155@tcp107 [8/256/0/180] [10690.739230] LNet: Added LNI 192.168.203.155@tcp108 [8/256/0/180] [10691.402456] LNet: Added LNI 192.168.203.155@tcp109 [8/256/0/180] [10692.065757] LNet: Added LNI 192.168.203.155@tcp110 [8/256/0/180] [10692.714795] LNet: Added LNI 192.168.203.155@tcp111 [8/256/0/180] [10693.332119] LNet: Added LNI 192.168.203.155@tcp112 [8/256/0/180] [10694.014227] LNet: Added LNI 192.168.203.155@tcp113 [8/256/0/180] [10694.608545] LNet: Added LNI 192.168.203.155@tcp114 [8/256/0/180] [10695.136553] LNet: Added LNI 192.168.203.155@tcp115 [8/256/0/180] [10695.759061] LNet: Added LNI 192.168.203.155@tcp116 [8/256/0/180] [10696.330176] LNet: Added LNI 192.168.203.155@tcp117 [8/256/0/180] [10696.922972] LNet: Added LNI 192.168.203.155@tcp118 [8/256/0/180] [10697.480046] LNet: Added LNI 192.168.203.155@tcp119 [8/256/0/180] [10698.054464] LNet: Added LNI 192.168.203.155@tcp120 [8/256/0/180] [10698.576923] LNet: Added LNI 192.168.203.155@tcp121 [8/256/0/180] [10699.186895] LNet: Added LNI 192.168.203.155@tcp122 [8/256/0/180] [10700.013413] LNet: Added LNI 192.168.203.155@tcp123 [8/256/0/180] [10700.653808] LNet: Added LNI 192.168.203.155@tcp124 [8/256/0/180] [10701.245285] LNet: Added LNI 192.168.203.155@tcp125 [8/256/0/180] [10701.858906] LNet: Added LNI 192.168.203.155@tcp126 [8/256/0/180] [10702.581401] LNet: Added LNI 192.168.203.155@tcp127 [8/256/0/180] [10703.243511] LNet: Added LNI 192.168.203.155@tcp128 [8/256/0/180] [10704.011990] LNet: Added LNI 192.168.203.155@tcp129 [8/256/0/180] [10704.795786] LNet: Added LNI 192.168.203.155@tcp130 [8/256/0/180] [10705.552811] LNet: Added LNI 192.168.203.155@tcp131 [8/256/0/180] [10706.261402] LNet: Added LNI 192.168.203.155@tcp132 [8/256/0/180] [10706.865146] LNet: Added LNI 192.168.203.155@tcp133 [8/256/0/180] [10707.516645] LNet: Added LNI 192.168.203.155@tcp134 [8/256/0/180] [10708.281480] LNet: Added LNI 192.168.203.155@tcp135 [8/256/0/180] [10713.378278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10713.415056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10713.423606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.504024] Lustre: 471740:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [10714.631220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10714.652032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10714.715051] Lustre: lustre-MDT0000: new disk, initializing [10714.814410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10714.823506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10718.159508] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10725.946550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10726.023545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10726.148758] Lustre: 471756:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10726.195954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10726.199221] Lustre: Skipped 1 previous similar message [10726.257866] Lustre: lustre-MDT0001: new disk, initializing [10726.279465] Lustre: 472537:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [10726.287600] Lustre: 472537:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10726.360288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10726.394158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10726.411566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10729.503335] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10733.088328] Lustre: 472953:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [10733.105723] Lustre: 472953:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10735.039589] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10737.725396] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10742.558499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10742.612291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10742.801315] Lustre: lustre-OST0000: new disk, initializing [10742.806987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10742.878386] Lustre: 473507:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [10742.890278] Lustre: 473507:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [10742.930718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10746.629129] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10751.067765] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10753.025453] Lustre: 474194:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [10753.034601] Lustre: 474194:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10753.087390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10753.096502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10753.154408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10758.629016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10758.629592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10758.632231] LustreError: Skipped 1 previous similar message [10758.633034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10758.643634] Lustre: Skipped 1 previous similar message [10761.404207] LustreError: 474340:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10761.591789] Lustre: server umount lustre-OST0000 complete [10767.328541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10767.339060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10767.351705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10767.358387] Lustre: Skipped 1 previous similar message [10768.867401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10768.868867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.660958] LustreError: 474542:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10770.664701] LustreError: 474542:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10770.713406] Lustre: server umount lustre-MDT0000 complete [10772.998898] LustreError: 471749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083786 with bad export cookie 5068568014930589604 [10773.000187] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10773.011067] LustreError: 471749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10773.106505] LustreError: 474743:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10773.113504] LustreError: 474743:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10773.215409] Lustre: server umount lustre-MDT0001 complete [10778.457166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10779.628545] Key type lgssc unregistered [10779.813671] LNet: 475282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10779.819600] LNetError: 475282:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10779.823469] LNet: Removed LNI 192.168.203.155@tcp [10780.831379] LNet: Removed LNI 192.168.203.155@tcp100 [10780.836898] LNetError: 475282:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10780.840262] LNetError: 475282:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10780.843376] LNet: Removed LNI 192.168.203.155@tcp101 [10781.855562] LNet: Removed LNI 192.168.203.155@tcp102 [10781.858982] LNetError: 475282:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10781.863966] LNetError: 475282:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10781.871133] LNet: Removed LNI 192.168.203.155@tcp103 [10781.874656] LNet: Removed LNI 192.168.203.155@tcp104 [10781.876952] LNet: Removed LNI 192.168.203.155@tcp105 [10781.879191] LNet: Removed LNI 192.168.203.155@tcp106 [10781.882248] LNet: Removed LNI 192.168.203.155@tcp107 [10781.884977] LNet: Removed LNI 192.168.203.155@tcp108 [10781.887207] LNet: Removed LNI 192.168.203.155@tcp109 [10781.889433] LNet: Removed LNI 192.168.203.155@tcp110 [10781.891451] LNet: Removed LNI 192.168.203.155@tcp111 [10781.893232] LNet: Removed LNI 192.168.203.155@tcp112 [10781.895214] LNet: Removed LNI 192.168.203.155@tcp113 [10781.897426] LNet: Removed LNI 192.168.203.155@tcp114 [10781.898689] LNet: Removed LNI 192.168.203.155@tcp115 [10781.900730] LNet: Removed LNI 192.168.203.155@tcp116 [10781.902814] LNet: Removed LNI 192.168.203.155@tcp117 [10781.905300] LNet: Removed LNI 192.168.203.155@tcp118 [10781.907205] LNet: Removed LNI 192.168.203.155@tcp119 [10781.908684] LNet: Removed LNI 192.168.203.155@tcp120 [10781.910064] LNet: Removed LNI 192.168.203.155@tcp121 [10781.911993] LNet: Removed LNI 192.168.203.155@tcp122 [10781.914209] LNet: Removed LNI 192.168.203.155@tcp123 [10781.917193] LNet: Removed LNI 192.168.203.155@tcp124 [10781.919393] LNet: Removed LNI 192.168.203.155@tcp125 [10781.921961] LNet: Removed LNI 192.168.203.155@tcp126 [10781.923510] LNet: Removed LNI 192.168.203.155@tcp127 [10781.925429] LNet: Removed LNI 192.168.203.155@tcp128 [10781.927380] LNet: Removed LNI 192.168.203.155@tcp129 [10781.929714] LNet: Removed LNI 192.168.203.155@tcp130 [10781.932252] LNet: Removed LNI 192.168.203.155@tcp131 [10781.934554] LNet: Removed LNI 192.168.203.155@tcp132 [10781.936373] LNet: Removed LNI 192.168.203.155@tcp133 [10781.938275] LNet: Removed LNI 192.168.203.155@tcp134 [10781.945758] LNet: Removed LNI 192.168.203.155@tcp135 [10782.335048] Key type .llcrypt unregistered [10782.336292] Key type ._llcrypt unregistered [10794.183185] Key type ._llcrypt registered [10794.185477] Key type .llcrypt registered [10794.236620] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10803.160231] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10803.781720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10803.792899] alg: No test for adler32 (adler32-zlib) [10804.699617] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [10804.850250] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10806.455202] Key type lgssc registered [10806.950905] Lustre: Echo OBD driver; http://www.lustre.org/ [10810.808140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10813.358682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10816.150744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10819.014591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10821.903090] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:10:34 (1769083834) [10828.960303] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10830.172462] Key type lgssc unregistered [10830.330284] LNet: 479295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10830.333174] LNetError: 479295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10830.344127] LNet: Removed LNI 192.168.203.155@tcp [10830.710709] Key type .llcrypt unregistered [10830.712372] Key type ._llcrypt unregistered [10838.729482] Key type ._llcrypt registered [10838.731099] Key type .llcrypt registered [10838.783678] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10839.201269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10839.246395] alg: No test for adler32 (adler32-zlib) [10840.157045] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [10840.343970] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10841.943152] Key type lgssc registered [10842.488252] Lustre: Echo OBD driver; http://www.lustre.org/ [10846.529540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10846.550194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10846.559839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10847.661257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10847.676935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10847.718189] Lustre: lustre-MDT0000: new disk, initializing [10847.757046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10847.764772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10849.251171] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10854.023544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10854.051529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10854.093792] Lustre: 480466:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10854.125602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10854.128358] Lustre: Skipped 1 previous similar message [10854.166313] Lustre: lustre-MDT0001: new disk, initializing [10854.208701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10854.225741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10854.230462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10855.824757] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10859.383866] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10860.760288] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10863.451208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10863.485412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10863.597358] Lustre: lustre-OST0000: new disk, initializing [10863.600263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10863.647672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10864.387973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10864.393251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10864.432544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10866.004212] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10869.564085] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10870.952484] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lnet_if_list [10872.949059] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10872.962588] LNet: Added LNI 192.168.203.155@tcp10 [8/256/0/180] [10875.039959] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10875.054814] LNet: Added LNI 192.168.203.155@tcp11 [8/256/0/180] [10876.495199] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10876.509678] LNet: Added LNI 192.168.203.155@tcp12 [8/256/0/180] [10878.009907] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10878.023822] LNet: Added LNI 192.168.203.155@tcp13 [8/256/0/180] [10879.644360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10879.656651] LNet: Added LNI 192.168.203.155@tcp14 [8/256/0/180] [10881.213168] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10881.226243] LNet: Added LNI 192.168.203.155@tcp15 [8/256/0/180] [10882.884484] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10882.898028] LNet: Added LNI 192.168.203.155@tcp16 [8/256/0/180] [10884.587506] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10884.603755] LNet: Added LNI 192.168.203.155@tcp17 [8/256/0/180] [10886.351370] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10886.364971] LNet: Added LNI 192.168.203.155@tcp18 [8/256/0/180] [10888.067653] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10888.080290] LNet: Added LNI 192.168.203.155@tcp19 [8/256/0/180] [10889.786106] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10889.799553] LNet: Added LNI 192.168.203.155@tcp20 [8/256/0/180] [10891.472122] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10891.485456] LNet: Added LNI 192.168.203.155@tcp21 [8/256/0/180] [10893.184572] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10893.197116] LNet: Added LNI 192.168.203.155@tcp22 [8/256/0/180] [10894.945275] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10894.957503] LNet: Added LNI 192.168.203.155@tcp23 [8/256/0/180] [10896.784955] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10896.798447] LNet: Added LNI 192.168.203.155@tcp24 [8/256/0/180] [10898.741552] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10898.755315] LNet: Added LNI 192.168.203.155@tcp25 [8/256/0/180] [10900.840503] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10900.856324] LNet: Added LNI 192.168.203.155@tcp26 [8/256/0/180] [10902.963622] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10902.978253] LNet: Added LNI 192.168.203.155@tcp27 [8/256/0/180] [10902.985543] LNetError: 479725:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10902.991232] LNetError: Protocol error connecting to 192.168.203.155@tcp26 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10904.980334] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10904.996526] LNet: Added LNI 192.168.203.155@tcp28 [8/256/0/180] [10905.000793] LNetError: 479728:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10905.005094] LNetError: Protocol error connecting to 192.168.203.155@tcp26 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10905.010779] LNetError: Skipped 1 previous similar message [10907.011221] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10907.023063] LNet: Added LNI 192.168.203.155@tcp29 [8/256/0/180] [10907.027513] LNetError: 479725:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10907.031783] LNetError: 479725:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [10907.034716] LNetError: Protocol error connecting to 192.168.203.155@tcp27 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10909.178717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10909.192598] LNet: Added LNI 192.168.203.155@tcp30 [8/256/0/180] [10909.197772] LNetError: 479727:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10909.202912] LNetError: Protocol error connecting to 192.168.203.155@tcp29 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10909.983087] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp29 [10911.318742] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10911.330662] LNet: Added LNI 192.168.203.155@tcp31 [8/256/0/180] [10913.366453] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10913.379214] LNet: Added LNI 192.168.203.155@tcp32 [8/256/0/180] [10913.383986] LNetError: 479725:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10913.387860] LNetError: 479725:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [10913.391408] LNetError: Protocol error connecting to 192.168.203.155@tcp30 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10913.396228] LNetError: Skipped 2 previous similar messages [10915.517117] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10915.531554] LNet: Added LNI 192.168.203.155@tcp33 [8/256/0/180] [10917.822817] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10917.836443] LNet: Added LNI 192.168.203.155@tcp34 [8/256/0/180] [10920.221169] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10920.235052] LNet: Added LNI 192.168.203.155@tcp35 [8/256/0/180] [10922.273975] LNetError: 479728:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10922.278140] LNetError: 479728:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [10922.281930] LNetError: Protocol error connecting to 192.168.203.155@tcp31 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10922.286964] LNetError: Skipped 8 previous similar messages [10922.633449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10922.648683] LNet: Added LNI 192.168.203.155@tcp36 [8/256/0/180] [10925.153063] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10925.166835] LNet: Added LNI 192.168.203.155@tcp37 [8/256/0/180] [10927.585122] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10927.598894] LNet: Added LNI 192.168.203.155@tcp38 [8/256/0/180] [10930.026035] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10930.039730] LNet: Added LNI 192.168.203.155@tcp39 [8/256/0/180] [10932.625163] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10932.641257] LNet: Added LNI 192.168.203.155@tcp40 [8/256/0/180] [10932.647760] LNetError: 479735:0:(peer.c:1886:lnet_peer_add_nid()) Peer 192.168.203.155@tcp NID 192.168.203.155@tcp is already locked with peer 192.168.203.155@tcp [10932.654051] LNetError: 479735:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.203.155@tcp has no available nets [10935.343814] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10935.360447] LNet: Added LNI 192.168.203.155@tcp41 [8/256/0/180] [10935.363058] LNetError: 479735:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.203.155@tcp has no available nets [10937.977889] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10937.989468] LNet: Added LNI 192.168.203.155@tcp42 [8/256/0/180] [10937.991261] LNetError: 479735:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.203.155@tcp has no available nets [10937.994619] LNetError: 479735:0:(lib-move.c:2321:lnet_handle_find_routed_path()) Skipped 1 previous similar message [10939.881608] LNetError: 496858:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10939.886159] LNetError: 496858:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 31 previous similar messages [10939.889730] LNetError: Protocol error connecting to 192.168.203.155@tcp26 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10939.894013] LNetError: Skipped 33 previous similar messages [10940.853444] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10940.866819] LNet: Added LNI 192.168.203.155@tcp43 [8/256/0/180] [10943.637310] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10943.652161] LNet: Added LNI 192.168.203.155@tcp44 [8/256/0/180] [10946.450316] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10946.464376] LNet: Added LNI 192.168.203.155@tcp45 [8/256/0/180] [10949.126283] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10949.139616] LNet: Added LNI 192.168.203.155@tcp46 [8/256/0/180] [10952.083407] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10952.095169] LNet: Added LNI 192.168.203.155@tcp47 [8/256/0/180] [10955.046947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10955.058776] LNet: Added LNI 192.168.203.155@tcp48 [8/256/0/180] [10957.837638] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10957.857507] LNet: Added LNI 192.168.203.155@tcp49 [8/256/0/180] [10960.768611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10960.780897] LNet: Added LNI 192.168.203.155@tcp50 [8/256/0/180] [10970.399416] LNet: 6 local NIs in recovery (showing 6): 192.168.203.155@tcp43, 192.168.203.155@tcp46, 192.168.203.155@tcp35, 192.168.203.155@tcp41, 192.168.203.155@tcp40, 192.168.203.155@tcp32 [10972.131134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10972.132449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10972.135755] LustreError: Skipped 1 previous similar message [10972.140944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10972.146513] Lustre: Skipped 1 previous similar message [10974.830066] LustreError: 504652:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10974.875345] Lustre: server umount lustre-OST0000 complete [10975.520427] LNetError: 479726:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [10975.524390] LNetError: 479726:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 50 previous similar messages [10975.528028] LNetError: Protocol error connecting to 192.168.203.155@tcp38 at host 192.168.203.155:988: is it running a compatible version of Lustre? [10975.531684] LNetError: Skipped 46 previous similar messages [10977.248488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10977.251206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10977.257782] Lustre: Skipped 2 previous similar messages [10977.261654] Lustre: Skipped 1 previous similar message [10982.368113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10982.371393] Lustre: Skipped 1 previous similar message [10982.468457] LustreError: 504853:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10982.471331] LustreError: 504853:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10982.506357] Lustre: server umount lustre-MDT0000 complete [10983.887302] LustreError: 480459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083997 with bad export cookie 8887289280424984557 [10983.890015] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10983.892434] LustreError: 480459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10983.970031] LustreError: 505054:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10983.974302] LustreError: 505054:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10984.041227] Lustre: server umount lustre-MDT0001 complete [10988.018058] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10989.179188] Key type lgssc unregistered [10989.357236] LNet: 505593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10989.361203] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10989.364107] LNet: Removed LNI 192.168.203.155@tcp [10990.367142] LNet: Removed LNI 192.168.203.155@tcp10 [10990.369126] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10990.371801] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10990.375139] LNet: Removed LNI 192.168.203.155@tcp11 [10991.391165] LNet: Removed LNI 192.168.203.155@tcp12 [10991.392884] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10991.396522] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10991.399541] LNet: Removed LNI 192.168.203.155@tcp13 [10992.415489] LNet: Removed LNI 192.168.203.155@tcp14 [10993.439141] LNet: Removed LNI 192.168.203.155@tcp15 [10993.440809] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10993.444242] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10993.447957] LNet: Removed LNI 192.168.203.155@tcp16 [10994.463152] LNet: Removed LNI 192.168.203.155@tcp17 [10995.487159] LNet: Removed LNI 192.168.203.155@tcp18 [10996.511203] LNet: Removed LNI 192.168.203.155@tcp19 [10997.535166] LNet: Removed LNI 192.168.203.155@tcp20 [10997.537100] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10997.540816] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10997.545239] LNet: Removed LNI 192.168.203.155@tcp21 [10998.559240] LNet: Removed LNI 192.168.203.155@tcp22 [10999.583196] LNet: Removed LNI 192.168.203.155@tcp23 [11000.607180] LNet: Removed LNI 192.168.203.155@tcp24 [11001.631476] LNet: Removed LNI 192.168.203.155@tcp25 [11002.655153] LNet: Removed LNI 192.168.203.155@tcp26 [11003.679162] LNet: Removed LNI 192.168.203.155@tcp27 [11004.703323] LNet: Removed LNI 192.168.203.155@tcp28 [11005.727174] LNet: Removed LNI 192.168.203.155@tcp29 [11005.728632] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11005.732160] LNetError: 505593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11005.735849] LNet: Removed LNI 192.168.203.155@tcp30 [11006.751178] LNet: Removed LNI 192.168.203.155@tcp31 [11006.752740] LNet: Removed LNI 192.168.203.155@tcp32 [11007.775212] LNet: Removed LNI 192.168.203.155@tcp33 [11008.799130] LNet: Removed LNI 192.168.203.155@tcp34 [11008.800922] LNet: Removed LNI 192.168.203.155@tcp35 [11009.823188] LNet: Removed LNI 192.168.203.155@tcp36 [11010.847141] LNet: Removed LNI 192.168.203.155@tcp37 [11011.871217] LNet: Removed LNI 192.168.203.155@tcp38 [11012.895173] LNet: Removed LNI 192.168.203.155@tcp39 [11013.919199] LNet: Removed LNI 192.168.203.155@tcp40 [11013.921180] LNet: Removed LNI 192.168.203.155@tcp41 [11014.943218] LNet: Removed LNI 192.168.203.155@tcp42 [11014.944866] LNet: Removed LNI 192.168.203.155@tcp43 [11015.967213] LNet: Removed LNI 192.168.203.155@tcp44 [11016.991182] LNet: Removed LNI 192.168.203.155@tcp45 [11016.993039] LNet: Removed LNI 192.168.203.155@tcp46 [11018.015247] LNet: Removed LNI 192.168.203.155@tcp47 [11019.039194] LNet: Removed LNI 192.168.203.155@tcp48 [11019.041471] LNet: Removed LNI 192.168.203.155@tcp49 [11019.049415] LNet: Removed LNI 192.168.203.155@tcp50 [11019.342876] Key type .llcrypt unregistered [11019.343955] Key type ._llcrypt unregistered [11027.501729] Key type ._llcrypt registered [11027.502805] Key type .llcrypt registered [11027.546364] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11027.977099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11028.001690] alg: No test for adler32 (adler32-zlib) [11028.869089] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [11028.963048] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11030.551127] Key type lgssc registered [11030.950184] Lustre: Echo OBD driver; http://www.lustre.org/ [11033.811915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11033.820832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.997682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11036.430224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11039.453192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11039.582377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11040.932920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11042.884717] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11044.281135] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11046.625216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11046.741886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11048.799126] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11050.769853] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11065.254840] loop0: detected capacity change from 0 to 204800000 [11065.379031] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11065.382683] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11065.387587] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11065.410218] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11067.854502] loop0: detected capacity change from 0 to 204800000 [11067.916764] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11067.921932] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11067.927783] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11067.988395] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11069.973903] loop0: detected capacity change from 0 to 204800000 [11071.993435] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11072.015401] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11072.059424] Lustre: fs15246-OST0000: new disk, initializing [11072.061712] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11072.080139] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11074.078324] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11078.286115] loop1: detected capacity change from 0 to 204800000 [11080.401072] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11080.423160] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11080.463829] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11080.488691] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11080.526989] Lustre: fs15246-MDT0000: new disk, initializing [11080.581302] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11080.592786] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11080.637662] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11081.696092] Lustre: 511213:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000004c3f05a x1855019068456960/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769084111 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11082.129193] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11090.759228] LustreError: 511213:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [11090.763233] LustreError: 511213:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [11090.767261] LustreError: 511213:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [11091.938823] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.943702] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11096.441164] LustreError: 512337:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [11096.465395] Lustre: server umount fs15246-MDT0000 complete [11097.060649] LustreError: 506813:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.067854] LustreError: 506813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11099.992195] LustreError: 512840:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [11099.995059] LustreError: 512840:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11100.010734] Lustre: server umount fs15246-OST0000 complete [11105.249444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11105.253260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.258845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.272490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.275412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.280820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11109.617085] LustreError: 513342:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11109.619616] LustreError: 513342:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [11109.665719] Lustre: server umount lustre-OST0000 complete [11111.391820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11111.394750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11111.400115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11111.402151] Lustre: Skipped 2 previous similar messages [11117.183671] LustreError: 513544:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11117.186294] LustreError: 513544:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11117.212109] Lustre: server umount lustre-MDT0000 complete [11117.535964] LustreError: 506818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.542900] LustreError: 506818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11118.575324] LustreError: 508437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769084131 with bad export cookie 18188120597563745996 [11118.579078] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11118.581565] LustreError: 508437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11118.708413] Lustre: server umount lustre-MDT0001 complete [11122.458196] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11123.493713] Key type lgssc unregistered [11123.628355] LNet: 514277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11123.632260] LNetError: 514277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11123.640274] LNet: Removed LNI 192.168.203.155@tcp [11123.948048] Key type .llcrypt unregistered [11123.949638] Key type ._llcrypt unregistered [11131.522489] Key type ._llcrypt registered [11131.523878] Key type .llcrypt registered [11131.571825] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11131.907390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11131.938582] alg: No test for adler32 (adler32-zlib) [11132.803645] Lustre: Lustre: Build Version: 2.17.50_4_g8fdf1cc [11132.890666] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11134.479152] Key type lgssc registered [11134.890874] Lustre: Echo OBD driver; http://www.lustre.org/ [11137.525455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11139.445926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11141.595714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11143.507985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11149.776217] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11153.410441] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11157.558916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11160.167064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11162.621460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11164.867506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11167.379724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11167.403466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11167.410375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11168.527589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11168.544840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11168.581893] Lustre: lustre-MDT0000: new disk, initializing [11168.608696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11168.617480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11170.173334] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11174.908771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11174.937082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11174.976117] Lustre: 518680:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11175.003308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11175.005507] Lustre: Skipped 1 previous similar message [11175.041905] Lustre: lustre-MDT0001: new disk, initializing [11175.076407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11175.091405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11175.095663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11176.497289] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11180.078474] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11181.454900] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11184.073895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11184.103313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11184.216435] Lustre: lustre-OST0000: new disk, initializing [11184.218880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11184.252782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11186.406873] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11190.023288] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11191.663553] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11191.814385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11191.818945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11191.857550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11192.769625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [11194.638920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11194.735456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11196.899365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11196.900300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11196.907874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11196.910208] LustreError: Skipped 1 previous similar message [11196.915130] Lustre: Skipped 1 previous similar message [11201.780121] LustreError: 521626:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11201.839919] Lustre: server umount lustre-OST0000 complete [11206.112348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11206.115928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.121156] Lustre: Skipped 1 previous similar message [11206.123360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11207.136779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11207.137065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11207.146506] Lustre: Skipped 1 previous similar message [11209.535816] LustreError: 521828:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11209.538501] LustreError: 521828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11209.563716] Lustre: server umount lustre-MDT0000 complete [11211.127881] LustreError: 518672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769084224 with bad export cookie 16218423879314143016 [11211.130513] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11211.132276] LustreError: 518672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11211.208362] LustreError: 522029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11211.210443] LustreError: 522029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11211.283817] Lustre: server umount lustre-MDT0001 complete [11220.596689] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11224.123440] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11227.750060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11230.350883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11232.779780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11235.331992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11235.985094] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11020 sec ========= 07:17:28 (1769084248) [11236.659523] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:17:28 (1769084248) === [11238.025461] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:17:30 (1769084250) === [11253.368836] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11254.519433] Key type lgssc unregistered [11254.686752] LNet: 526877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11254.692748] LNetError: 526877:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11254.703856] LNet: Removed LNI 192.168.203.155@tcp [11255.021557] Key type .llcrypt unregistered [11255.022886] Key type ._llcrypt unregistered