[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490586924 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003363] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008369] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011123] LSM: Security Framework initializing [ 0.012044] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.021849] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026294] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030030] *** VALIDATE tmpfs *** [ 0.031452] *** VALIDATE proc *** [ 0.032213] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035239] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038021] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042260] debug: unmapping init [mem 0xffffffffa5859000-0xffffffffa5860fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045732] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046018] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053305] rcu: Hierarchical SRCU implementation. [ 0.055425] smp: Bringing up secondary CPUs ... [ 0.056649] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.062336] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148608] node 0 deferred pages initialised in 81ms [ 0.150156] devtmpfs: initialized [ 0.151251] x86/mm: Memory block size: 128MB [ 0.154149] gcov: version magic: 0x41383552 [ 0.157235] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161170] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163279] pinctrl core: initialized pinctrl subsystem [ 0.165144] [ 0.165669] ************************************************************* [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ** ** [ 0.172014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174011] ** ** [ 0.176012] ** This means that this kernel is built to expose internal ** [ 0.179013] ** IOMMU data structures, which may compromise security on ** [ 0.182012] ** your system. ** [ 0.184010] ** ** [ 0.186013] ** If you see this message and you are not debugging the ** [ 0.188018] ** kernel, report this immediately to your vendor! ** [ 0.191013] ** ** [ 0.193011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195013] ************************************************************* [ 0.198646] NET: Registered protocol family 16 [ 0.200473] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209104] cpuidle: using governor menu [ 0.211361] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213421] PCI: Using configuration type 1 for base access [ 0.215122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225178] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227167] cryptd: max_cpu_qlen set to 1000 [ 0.229423] ACPI: Added _OSI(Module Device) [ 0.230034] ACPI: Added _OSI(Processor Device) [ 0.231015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237159] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243615] ACPI: Interpreter enabled [ 0.245064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246009] ACPI: Using IOAPIC for interrupt routing [ 0.248123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250267] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258901] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272323] acpiphp: Slot [2] registered [ 0.274122] acpiphp: Slot [5] registered [ 0.275149] acpiphp: Slot [6] registered [ 0.277192] acpiphp: Slot [7] registered [ 0.279141] acpiphp: Slot [8] registered [ 0.280125] acpiphp: Slot [9] registered [ 0.282093] acpiphp: Slot [10] registered [ 0.283117] acpiphp: Slot [3] registered [ 0.285135] acpiphp: Slot [4] registered [ 0.286111] acpiphp: Slot [11] registered [ 0.287104] acpiphp: Slot [12] registered [ 0.289097] acpiphp: Slot [13] registered [ 0.290075] acpiphp: Slot [14] registered [ 0.291116] acpiphp: Slot [15] registered [ 0.293115] acpiphp: Slot [16] registered [ 0.295123] acpiphp: Slot [17] registered [ 0.296117] acpiphp: Slot [18] registered [ 0.298107] acpiphp: Slot [19] registered [ 0.299065] acpiphp: Slot [20] registered [ 0.299921] acpiphp: Slot [21] registered [ 0.301095] acpiphp: Slot [22] registered [ 0.302075] acpiphp: Slot [23] registered [ 0.303086] acpiphp: Slot [24] registered [ 0.304095] acpiphp: Slot [25] registered [ 0.306099] acpiphp: Slot [26] registered [ 0.307093] acpiphp: Slot [27] registered [ 0.308096] acpiphp: Slot [28] registered [ 0.310123] acpiphp: Slot [29] registered [ 0.311112] acpiphp: Slot [30] registered [ 0.313093] acpiphp: Slot [31] registered [ 0.314069] PCI host bridge to bus 0000:00 [ 0.315016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.322025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332969] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335416] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345838] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.350992] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362776] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364884] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371236] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400530] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446037] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485106] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.494016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.526015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.538540] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.546011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.553016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.577013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.589216] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.595024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.604017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.624014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.635752] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.642016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.648017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.667016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.679362] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.682420] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.684392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.686433] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.688259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.695098] iommu: Default domain type: Passthrough [ 0.697486] SCSI subsystem initialized [ 0.699138] ACPI: bus type USB registered [ 0.700115] usbcore: registered new interface driver usbfs [ 0.702085] usbcore: registered new interface driver hub [ 0.704078] usbcore: registered new device driver usb [ 0.707179] pps_core: LinuxPPS API ver. 1 registered [ 0.708009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.711053] PTP clock support registered [ 0.713147] EDAC MC: Ver: 3.0.0 [ 0.716213] PCI: Using ACPI for IRQ routing [ 0.719243] NetLabel: Initializing [ 0.720019] NetLabel: domain hash size = 128 [ 0.722018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.724140] NetLabel: unlabeled traffic allowed by default [ 0.727110] vgaarb: loaded [ 0.728328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.730019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.736282] clocksource: Switched to clocksource kvm-clock [ 0.843645] VFS: Disk quotas dquot_6.6.0 [ 0.845895] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.848524] *** VALIDATE ramfs *** [ 0.849860] *** VALIDATE hugetlbfs *** [ 0.851538] pnp: PnP ACPI init [ 0.853946] pnp: PnP ACPI: found 6 devices [ 0.884284] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.887578] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.890097] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.892617] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.895352] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.898230] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.901443] NET: Registered protocol family 2 [ 0.904432] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.909511] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.913483] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.917938] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.920989] TCP: Hash tables configured (established 65536 bind 65536) [ 0.923507] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.925989] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.928222] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.930639] NET: Registered protocol family 1 [ 0.933374] RPC: Registered named UNIX socket transport module. [ 0.936099] RPC: Registered udp transport module. [ 0.937837] RPC: Registered tcp transport module. [ 0.939603] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.942400] NET: Registered protocol family 44 [ 0.943836] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.946105] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.948331] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.950685] PCI: CLS 0 bytes, default 64 [ 0.952485] Unpacking initramfs... [ 2.352170] debug: unmapping init [mem 0xffff97787cc54000-0xffff97787ffbffff] [ 2.357868] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.360379] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.363664] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.882862] Initialise system trusted keyrings [ 2.884799] Key type blacklist registered [ 2.886995] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.896455] zbud: loaded [ 2.899405] *** VALIDATE nfs *** [ 2.900730] *** VALIDATE nfs4 *** [ 2.903041] pstore: using deflate compression [ 2.906743] Platform Keyring initialized [ 3.014423] NET: Registered protocol family 38 [ 3.016512] Key type asymmetric registered [ 3.018116] Asymmetric key parser 'x509' registered [ 3.019920] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.023572] io scheduler mq-deadline registered [ 3.025208] io scheduler kyber registered [ 3.026791] io scheduler bfq registered [ 3.028489] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.031196] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.033800] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.036694] ACPI: Power Button [PWRF] [ 3.043451] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.052210] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.068670] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.075260] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.090799] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.119014] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.148901] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.154407] Non-volatile memory driver v1.3 [ 3.156428] Linux agpgart interface v0.103 [ 3.190201] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.193115] vda: detected capacity change from 0 to 74825728 [ 3.209495] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.220603] vdb: detected capacity change from 0 to 1073741824 [ 3.246605] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.250675] vdc: detected capacity change from 0 to 2621440000 [ 3.269693] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.272152] vdd: detected capacity change from 0 to 2621440000 [ 3.288384] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.290829] vde: detected capacity change from 0 to 4294967296 [ 3.306644] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.310816] vdf: detected capacity change from 0 to 4294967296 [ 3.317829] libphy: Fixed MDIO Bus: probed [ 3.323351] usbcore: registered new interface driver usbserial_generic [ 3.325606] usbserial: USB Serial support registered for generic [ 3.328716] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.333500] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.335184] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.337777] mousedev: PS/2 mouse device common for all mice [ 3.340635] rtc_cmos 00:05: RTC can wake from S4 [ 3.343969] rtc_cmos 00:05: registered as rtc0 [ 3.345850] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.348873] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.351677] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.353099] intel_pstate: CPU model not supported [ 3.359283] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.364118] hid: raw HID events driver (C) Jiri Kosina [ 3.366091] usbcore: registered new interface driver usbhid [ 3.367968] usbhid: USB HID core driver [ 3.369546] drop_monitor: Initializing network drop monitor service [ 3.372126] Initializing XFRM netlink socket [ 3.374326] NET: Registered protocol family 10 [ 3.377912] Segment Routing with IPv6 [ 3.379629] NET: Registered protocol family 17 [ 3.381783] mpls_gso: MPLS GSO support [ 3.387855] RAS: Correctable Errors collector initialized. [ 3.390014] AVX version of gcm_enc/dec engaged. [ 3.391817] AES CTR mode by8 optimization enabled [ 3.474688] sched_clock: Marking stable (3474648434, 0)->(4371782710, -897134276) [ 3.478653] registered taskstats version 1 [ 3.482839] Loading compiled-in X.509 certificates [ 3.485313] zswap: loaded using pool lzo/zbud [ 3.510938] Key type big_key registered [ 3.523767] Key type encrypted registered [ 3.525347] ima: No TPM chip found, activating TPM-bypass! [ 3.526856] ima: Allocated hash algorithm: sha1 [ 3.528641] ima: No architecture policies found [ 3.530106] evm: Initialising EVM extended attributes: [ 3.531538] evm: security.selinux [ 3.532477] evm: security.ima [ 3.533329] evm: security.capability [ 3.534380] evm: HMAC attrs: 0x1 [ 3.536722] rtc_cmos 00:05: setting system clock to 2026-08-24 23:17:54 UTC (1787613474) [ 3.542185] debug: unmapping init [mem 0xffffffffa6803000-0xffffffffa69fffff] [ 3.546018] debug: unmapping init [mem 0xffffffffa5582000-0xffffffffa5858fff] [ 3.555055] Write protecting the kernel read-only data: 28672k [ 3.558407] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 3.561051] debug: unmapping init [mem 0xffffffffa4514000-0xffffffffa45fffff] [ 3.597242] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.605733] systemd[1]: Detected virtualization kvm. [ 3.607810] systemd[1]: Detected architecture x86-64. [ 3.609733] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.634756] systemd[1]: No hostname configured. [ 3.637017] systemd[1]: Set hostname to . [ 3.639449] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642438] systemd[1]: Initializing machine ID from random generator. [ 3.759249] random: systemd: uninitialized urandom read (16 bytes read) [ 3.762436] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.766271] random: systemd: uninitialized urandom read (16 bytes read) [ 3.769919] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.775827] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.477490] device-mapper: uevent: version 1.0.3 [ 4.480757] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 5.217305] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.286457] virtio_net virtio0 ens2: renamed from eth0 [ 5.345633] scsi host0: ata_piix [ 5.354516] scsi host1: ata_piix [ 5.356364] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.359204] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.231986] dracut-initqueue[576]: RTNETLINK answers: File exists [ 10.039961] random: crng init done [ 10.041594] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.571886] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.169476] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.972072] SELinux: Disabled at runtime. [ 13.108470] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.122756] systemd[1]: Detected virtualization kvm. [ 13.124910] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.739320] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.764225] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.792685] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.813333] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.831520] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.851180] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.869719] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ 15.216139] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.360589] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.326461] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.398020] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.669149] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.733957] EDAC sbridge: Ver: 1.1.2 [ 21.537460] Key type dns_resolver registered [ 22.028178] NFS: Registering the id_resolver key type [ 22.031484] Key type id_resolver registered [ 22.034529] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg620-server login: [ 36.518741] hrtimer: interrupt took 14053676 ns [ 86.073546] libcfs: loading out-of-tree module taints kernel. [ 86.150911] Key type ._llcrypt registered [ 86.156195] Key type .llcrypt registered [ 86.302057] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 105.990907] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 106.992742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.999969] alg: No test for adler32 (adler32-zlib) [ 108.472458] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 109.771220] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 111.520875] Key type lgssc registered [ 113.269956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.106135] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 203.809961] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 217.995535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 218.055761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.333721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 219.356339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 219.436457] Lustre: lustre-MDT0000: new disk, initializing [ 219.490219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.509392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.729457] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.722859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.851407] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 239.913208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 239.923983] Lustre: Skipped 1 previous similar message [ 240.015526] Lustre: lustre-MDT0001: new disk, initializing [ 240.079143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.116959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 240.128410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 244.541477] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.409061] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.506036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.803634] Lustre: lustre-OST0000: new disk, initializing [ 257.807848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 257.812643] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.862460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.182361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 260.195402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 260.269413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.605034] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.962935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.092838] Lustre: lustre-OST0001: new disk, initializing [ 278.098449] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 278.108169] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 278.174760] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 284.135129] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.760342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.769316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.854530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 297.536599] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 303.517698] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 310.748472] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing check_logdir /tmp/testlogs/ [ 315.624023] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing yml_node [ 320.725673] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 323.113411] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 326.185540] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 328.532282] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 19:23:18 EDT 2026 [ 346.425440] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 348.233820] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 364.513640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.526244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.526509] Lustre: Skipped 1 previous similar message [ 364.544658] Lustre: Skipped 3 previous similar messages [ 369.634113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.646732] Lustre: Skipped 3 previous similar messages [ 371.008737] Lustre: server umount lustre-MDT0000 complete [ 374.833941] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613845 with bad export cookie 517552107097971192 [ 374.840673] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.847947] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 379.872061] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.874553] LustreError: 7750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.874959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.888350] Lustre: Skipped 3 previous similar messages [ 379.918147] LustreError: 7750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 381.245887] Lustre: server umount lustre-MDT0001 complete [ 384.901460] Lustre: server umount lustre-OST0000 complete [ 388.490859] Lustre: server umount lustre-OST0001 complete [ 395.180620] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 404.597636] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 443.936929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.197340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.249392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.361479] Lustre: lustre-MDT0000: new disk, initializing [ 444.483313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.511533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 447.975059] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 460.417681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.601418] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.710307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.715332] Lustre: Skipped 1 previous similar message [ 460.852437] Lustre: lustre-MDT0001: new disk, initializing [ 461.033982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.130498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.152639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 466.151541] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 476.982989] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 482.718097] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 490.518919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.688302] Lustre: lustre-OST0000: new disk, initializing [ 490.691289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 490.695774] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 490.783207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.888862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 491.907861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 492.111963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 498.532961] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 509.879830] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.522537] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 516.811701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 521.962954] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 522.132937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 527.842203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.856321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.874609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.883374] Lustre: Skipped 2 previous similar messages [ 530.254526] Lustre: server umount lustre-OST0000 complete [ 538.080634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.090114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.092333] LustreError: Skipped 1 previous similar message [ 538.099584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.123928] Lustre: Skipped 2 previous similar messages [ 540.713726] Lustre: server umount lustre-MDT0000 complete [ 543.202732] LustreError: 15084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.217862] LustreError: 15084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 544.512495] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614015 with bad export cookie 517552107097972725 [ 544.525155] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.534824] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 544.854408] Lustre: server umount lustre-MDT0001 complete [ 549.967373] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:26:59 (1787614019) [ 552.418685] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 554.511431] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:27:04 (1787614024) [ 562.047450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.650396] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.750171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.200953] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 567.781110] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.904581] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.344883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.510980] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 590.874157] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 597.225472] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 604.153837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.854658] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 619.036789] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 633.313206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 633.314848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.318566] LustreError: Skipped 1 previous similar message [ 633.329935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 633.339076] Lustre: Skipped 1 previous similar message [ 633.361115] Lustre: Skipped 2 previous similar messages [ 636.948581] Lustre: server umount lustre-OST0000 complete [ 643.560137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.568879] Lustre: Skipped 1 previous similar message [ 653.793604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.808672] Lustre: Skipped 5 previous similar messages [ 655.328092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 655.564902] Lustre: server umount lustre-MDT0000 complete [ 658.919143] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.946812] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 659.417670] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614130 with bad export cookie 517552107097973607 [ 659.420546] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.440886] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 659.900978] Lustre: server umount lustre-MDT0001 complete [ 673.257200] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 676.564975] Key type lgssc unregistered [ 676.905875] LNet: 22996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.918093] LNetError: 22996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.940875] LNet: Removed LNI 192.168.206.120@tcp [ 677.883264] Key type .llcrypt unregistered [ 677.885238] Key type ._llcrypt unregistered [ 704.373706] Key type ._llcrypt registered [ 704.376353] Key type .llcrypt registered [ 704.520784] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 705.752990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 705.775130] alg: No test for adler32 (adler32-zlib) [ 706.937696] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 707.259399] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 708.927187] Key type lgssc registered [ 710.193193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 719.993232] loop: module loaded [ 720.078275] loop0: detected capacity change from 0 to 204800000 [ 720.318771] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.350623] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.767985] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.551270] loop0: detected capacity change from 0 to 204800000 [ 726.772570] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.787348] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.793560] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 732.068783] loop0: detected capacity change from 0 to 204800000 [ 738.071717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 738.102542] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.407820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 739.452492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 739.547103] Lustre: lustre-MDT0000: new disk, initializing [ 739.661234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.695635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 744.036944] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 753.125179] loop1: detected capacity change from 0 to 204800000 [ 758.754800] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 759.087375] Lustre: lustre-OST07c6: new disk, initializing [ 759.091935] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 759.094529] Lustre: Skipped 1 previous similar message [ 759.100453] Lustre: 26187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 759.188426] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 765.627942] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 770.345759] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 770.359827] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 770.366560] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 770.373160] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 770.378636] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 770.384050] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 771.579505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 771.614265] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 771.670816] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 787.141639] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 787.159717] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 787.180406] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 787.200371] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 787.215686] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 787.228489] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.379302] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 792.397675] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 792.408483] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.415211] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.420843] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.424203] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.436090] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 792.444179] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.447069] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 792.451491] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.454920] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.457698] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.642097] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 795.654235] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 795.658869] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 795.667246] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.679840] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 795.688198] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.698189] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 795.717268] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.728921] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 795.740022] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 795.751488] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 795.761858] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.154915] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 800.160132] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 800.164329] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 800.173609] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.178712] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 800.182969] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.189086] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 800.195563] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.203120] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 800.211632] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.218170] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 800.223396] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.345142] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 811.368758] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 811.386742] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.393499] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 811.401279] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.409283] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 811.413347] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 811.419435] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 811.428052] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 811.431889] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 811.438049] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.444712] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 863.711923] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 863.725127] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.741931] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 867.188481] Lustre: server umount lustre-OST07c6 complete [ 877.666351] Lustre: server umount lustre-MDT0000 complete [ 894.690401] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 897.442382] Key type lgssc unregistered [ 897.721889] LNet: 28749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.741912] LNetError: 28749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.764735] LNet: Removed LNI 192.168.206.120@tcp [ 898.600474] Key type .llcrypt unregistered [ 898.602696] Key type ._llcrypt unregistered [ 918.041720] Key type ._llcrypt registered [ 918.051546] Key type .llcrypt registered [ 918.149847] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 919.216119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 919.325783] alg: No test for adler32 (adler32-zlib) [ 920.380519] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 920.607240] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 922.239198] Key type lgssc registered [ 923.513200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 931.419412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 931.441160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.909793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.651141] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 942.050124] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.062881] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 945.849681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.167179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 950.153875] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 957.626469] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 963.206908] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 969.976088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.316707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.574025] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 977.463171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 984.655899] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1001.954206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1001.959332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.971968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1002.985124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.996431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.015155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1003.027427] Lustre: Skipped 1 previous similar message [ 1006.665692] Lustre: server umount lustre-OST0000 complete [ 1012.707083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1012.724885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.755907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.570849] Lustre: server umount lustre-MDT0000 complete [ 1018.345390] LustreError: 30021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.366491] LustreError: 30021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1020.493584] LustreError: 30002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614491 with bad export cookie 642502023749541022 [ 1020.506557] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.508465] LustreError: 30002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1020.907488] Lustre: server umount lustre-MDT0001 complete [ 1032.158532] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1034.515540] Key type lgssc unregistered [ 1034.749911] LNet: 33762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.757058] LNetError: 33762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.789221] LNet: Removed LNI 192.168.206.120@tcp [ 1035.620590] Key type .llcrypt unregistered [ 1035.623811] Key type ._llcrypt unregistered [ 1055.495453] Key type ._llcrypt registered [ 1055.499750] Key type .llcrypt registered [ 1055.628173] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1057.056835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.148773] alg: No test for adler32 (adler32-zlib) [ 1058.190713] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1058.475604] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1060.217952] Key type lgssc registered [ 1061.681706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1069.411711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1069.434239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.806778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.134928] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1080.294445] LustreError: 35033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.313851] LustreError: 35033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1083.117821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.634948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.838900] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1095.515886] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.677735] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.730652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.215098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.314658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1116.307903] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1124.370503] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1139.169993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1139.177810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.189855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.210971] Lustre: Skipped 1 previous similar message [ 1144.305132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.314082] Lustre: Skipped 1 previous similar message [ 1144.473259] Lustre: server umount lustre-OST0000 complete [ 1149.410518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.415047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.427417] Lustre: Skipped 1 previous similar message [ 1149.444397] Lustre: Skipped 1 previous similar message [ 1154.528269] LustreError: 35760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.537555] LustreError: 35760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1154.557443] Lustre: server umount lustre-MDT0000 complete [ 1157.977243] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614628 with bad export cookie 18335800008651135426 [ 1157.980448] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.001587] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1158.294186] Lustre: server umount lustre-MDT0001 complete [ 1169.007891] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1171.759819] Key type lgssc unregistered [ 1171.997925] LNet: 38589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.006756] LNetError: 38589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.020635] LNet: Removed LNI 192.168.206.120@tcp [ 1172.793840] Key type .llcrypt unregistered [ 1172.802328] Key type ._llcrypt unregistered [ 1197.209643] Key type ._llcrypt registered [ 1197.215261] Key type .llcrypt registered [ 1197.338547] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1198.592630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.626777] alg: No test for adler32 (adler32-zlib) [ 1199.704229] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1199.926734] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1201.631160] Key type lgssc registered [ 1202.772435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1210.051920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1210.071557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.413665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.684878] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1221.600502] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.624586] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1223.723305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.086821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.508489] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1237.148587] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1243.189570] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1249.587833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1249.935292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.348852] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1257.066687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1261.938366] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1273.827961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.839121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.841729] Lustre: Skipped 2 previous similar messages [ 1273.868986] Lustre: Skipped 2 previous similar messages [ 1275.057265] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.20@tcp (stopping) [ 1278.275471] Lustre: server umount lustre-MDT0000 complete [ 1278.950459] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.972090] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1280.145835] LustreError: 42495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.047775] LustreError: 39987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614753 with bad export cookie 11041857795000993155 [ 1282.058071] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.061973] LustreError: 39987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1284.065976] LustreError: 40001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.067154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.107907] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1287.073763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1287.082476] Lustre: Skipped 1 previous similar message [ 1290.382782] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.420075] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1292.199530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1292.203977] Lustre: Skipped 1 previous similar message [ 1296.351977] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1296.675996] Lustre: server umount lustre-MDT0001 complete [ 1311.923310] Lustre: server umount lustre-OST0000 complete [ 1329.925635] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1332.917724] Key type lgssc unregistered [ 1333.253735] LNet: 43870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.269327] LNetError: 43870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.312317] LNet: Removed LNI 192.168.206.120@tcp [ 1334.527790] Key type .llcrypt unregistered [ 1334.530574] Key type ._llcrypt unregistered [ 1360.196866] Key type ._llcrypt registered [ 1360.199740] Key type .llcrypt registered [ 1360.299685] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1361.468848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.524835] alg: No test for adler32 (adler32-zlib) [ 1362.593366] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1362.888493] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1364.551209] Key type lgssc registered [ 1365.678399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1373.074557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1373.086638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.502922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.978598] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1384.931524] LustreError: 45287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.952973] LustreError: 45287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1387.480614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.770988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1391.648338] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1398.577653] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.837861] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1411.789646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1412.256176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.620243] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1419.438345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1426.290744] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1434.080931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.093729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.114416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.105860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1435.109394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.116888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.154722] Lustre: Skipped 2 previous similar messages [ 1439.012992] Lustre: server umount lustre-OST0000 complete [ 1459.680468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1459.686055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.699056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.989760] Lustre: server umount lustre-MDT0000 complete [ 1465.824356] LustreError: 45283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.852963] LustreError: 45283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1466.591514] LustreError: 45268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614937 with bad export cookie 14988528151305375970 [ 1466.602638] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.135914] Lustre: server umount lustre-MDT0001 complete [ 1484.927424] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1487.694857] Key type lgssc unregistered [ 1487.988770] LNet: 48997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.999628] LNetError: 48997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.063691] LNet: Removed LNI 192.168.206.120@tcp [ 1490.037269] Key type .llcrypt unregistered [ 1490.042725] Key type ._llcrypt unregistered [ 1513.980762] Key type ._llcrypt registered [ 1513.984852] Key type .llcrypt registered [ 1514.137251] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1515.258510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.273109] alg: No test for adler32 (adler32-zlib) [ 1516.292540] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1516.579321] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1518.247258] Key type lgssc registered [ 1519.328727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1526.499896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1526.510682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.001166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.600399] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1537.508420] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.517892] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1542.425561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.626638] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.917693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1548.003769] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1556.602877] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1562.490786] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1569.439966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1569.780392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.884071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1577.424102] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1584.885748] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1591.112617] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1594.531791] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1606.525832] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2419h44m36s [ 1607.805481] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1609.696752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1609.705892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.723171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.790181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.795903] Lustre: Skipped 4 previous similar messages [ 1612.968625] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.20@tcp (stopping) [ 1615.641966] Lustre: server umount lustre-MDT0000 complete [ 1616.872416] LustreError: 51408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.890781] LustreError: 51408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1618.998205] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1619.083119] LustreError: 50409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.237528] LustreError: 52415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.256730] LustreError: 52415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1625.771598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.915279] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1626.215113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.282725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1627.262702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1631.487930] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1635.455826] Lustre: DEBUG MARKER: Wait for df (36738) ... [ 1637.143300] Lustre: DEBUG MARKER: done [ 1641.445301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1641.456674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.470214] Lustre: Skipped 2 previous similar messages [ 1641.485247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1641.489856] Lustre: Skipped 1 previous similar message [ 1642.978979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.098602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1648.106981] Lustre: Skipped 3 previous similar messages [ 1655.263252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1655.501944] Lustre: server umount lustre-OST0000 complete [ 1663.460824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1663.463782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.480185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.501064] Lustre: Skipped 1 previous similar message [ 1663.510291] Lustre: Skipped 3 previous similar messages [ 1665.530494] Lustre: server umount lustre-MDT0000 complete [ 1669.467640] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615140 with bad export cookie 3883375902185106188 [ 1669.472510] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1669.488776] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1669.888144] Lustre: server umount lustre-MDT0001 complete [ 1681.377502] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1684.451866] Key type lgssc unregistered [ 1684.828900] LNet: 55344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.837194] LNetError: 55344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.868429] LNet: Removed LNI 192.168.206.120@tcp [ 1685.693122] Key type .llcrypt unregistered [ 1685.695348] Key type ._llcrypt unregistered [ 1721.034261] Key type ._llcrypt registered [ 1721.036631] Key type .llcrypt registered [ 1721.160929] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1722.374924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.400913] alg: No test for adler32 (adler32-zlib) [ 1723.495898] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1723.732856] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1725.383398] Key type lgssc registered [ 1726.407712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1733.631219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1733.650740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.907518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1734.930447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1735.075299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.478459] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1747.611236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.814385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1747.854198] Lustre: 57533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1748.106207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1752.974145] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1760.773725] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1766.719898] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1773.274318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.410740] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1773.565103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.813785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1779.361528] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1787.029440] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1794.688797] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2419h47m44s [ 1796.424918] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1803.476446] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1806.016699] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1806.051712] LustreError: 59234:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1826.500554] Lustre: lustre-MDT0000: Client c327607a-91a7-422a-989f-282c0d0890dd (at 192.168.206.20@tcp) reconnecting [ 1846.055701] LustreError: 59234:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1848.906292] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1851.349635] Lustre: DEBUG MARKER: done [ 1860.576113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1860.584627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.600506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.088146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.099336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.116449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.133559] Lustre: Skipped 1 previous similar message [ 1866.217851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.231305] Lustre: Skipped 1 previous similar message [ 1871.341473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1871.348708] Lustre: Skipped 1 previous similar message [ 1872.351305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1872.464615] Lustre: server umount lustre-OST0000 complete [ 1876.453702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1876.456429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.468101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.473461] Lustre: Skipped 1 previous similar message [ 1888.746569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.751971] Lustre: Skipped 6 previous similar messages [ 1890.271459] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1890.479714] Lustre: server umount lustre-MDT0000 complete [ 1893.741287] LustreError: 57523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615364 with bad export cookie 11335147123445936450 [ 1893.754823] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1894.038384] Lustre: server umount lustre-MDT0001 complete [ 1904.027471] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1906.821360] Key type lgssc unregistered [ 1907.102582] LNet: 61866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.111055] LNetError: 61866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.133110] LNet: Removed LNI 192.168.206.120@tcp [ 1908.018825] Key type .llcrypt unregistered [ 1908.024205] Key type ._llcrypt unregistered [ 1953.712359] Key type ._llcrypt registered [ 1953.717467] Key type .llcrypt registered [ 1953.846547] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1955.198037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.238412] alg: No test for adler32 (adler32-zlib) [ 1956.267912] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 1956.500824] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1958.135160] Key type lgssc registered [ 1959.194086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.082246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.115495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.329937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1963.353597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1963.483584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.231604] Lustre: Failing over lustre-MDT0000 [ 1964.435515] Lustre: server umount lustre-MDT0000 complete [ 1970.458563] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:50:40 (1787615440) [ 1976.042832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.537507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.343355] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1991.160635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.300326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1991.355951] Lustre: 64666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1991.714345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1996.207537] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2003.414835] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2008.699293] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2015.487389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2015.631730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2015.830122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.475143] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2023.059609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2028.353125] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2033.342467] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2042.849077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2042.859807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.877818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.873397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.875992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.885188] Lustre: Skipped 1 previous similar message [ 2043.898501] Lustre: Skipped 2 previous similar messages [ 2045.715861] Lustre: server umount lustre-MDT0000 complete [ 2049.472169] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615520 with bad export cookie 17049950780934610738 [ 2049.476069] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.495161] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2049.798069] Lustre: server umount lustre-MDT0001 complete [ 2053.184305] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2057.530522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2065.807583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.495367] Lustre: 63592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787615525/real 1787615525] req@00000000502edee6 x1874450634023168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787615541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.531948] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.657245] LustreError: 63589:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000049014056 x1874450634023936/t0(0) o250->MGC192.168.206.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.871273] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.892450] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2074.994119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.129144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2075.942747] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787615530/real 1787615530] req@000000008e53d5a8 x1874450634023296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787615546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2076.003702] LustreError: 66515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.765876] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2080.045625] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.058582] Lustre: 63590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787615535/real 1787615535] req@00000000c7cffe5b x1874450634023552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787615551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2085.152049] Lustre: 63591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787615540/real 1787615540] req@0000000088c1e848 x1874450634023680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787615556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2086.382613] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.410691] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2088.963900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.259725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2093.587635] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2094.588786] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2099.754486] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2100.459887] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2105.846932] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2113.318169] Lustre: server umount lustre-MDT0000 complete [ 2115.042512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2115.047562] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.051720] LustreError: 68440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.096310] Lustre: Skipped 2 previous similar messages [ 2117.039099] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615587 with bad export cookie 17049950780934613013 [ 2117.050485] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.056528] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2117.475682] Lustre: server umount lustre-MDT0001 complete [ 2123.834683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2133.119566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.287226] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787615591/real 1787615591] req@00000000f44a43d3 x1874450634049024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787615607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.311132] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.896680] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.914934] LustreError: 66514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2142.997419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.076477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2147.485935] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2148.130853] Lustre: 63592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787615603/real 1787615603] req@0000000004891d62 x1874450634049664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787615619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2148.181461] Lustre: 63592:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2155.493717] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2156.022405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.103627] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2161.640866] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2168.839164] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2174.538217] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.119482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.122813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.122980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.140077] Lustre: Skipped 1 previous similar message [ 2182.668296] Lustre: server umount lustre-MDT0000 complete [ 2185.743297] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615656 with bad export cookie 17049950780934614392 [ 2185.746754] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.754822] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2186.153496] Lustre: server umount lustre-MDT0001 complete [ 2192.141591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2195.569269] Lustre: DEBUG MARKER: files compared the same [ 2207.447451] Lustre: server umount lustre-OST0000 complete [ 2224.989662] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 2228.087514] Key type lgssc unregistered [ 2228.375825] LNet: 74397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.393214] LNetError: 74397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.407868] LNet: Removed LNI 192.168.206.120@tcp [ 2229.198507] Key type .llcrypt unregistered [ 2229.205547] Key type ._llcrypt unregistered [ 2254.495411] Key type ._llcrypt registered [ 2254.497890] Key type .llcrypt registered [ 2254.663045] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2255.707068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.789518] alg: No test for adler32 (adler32-zlib) [ 2256.996432] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2257.287231] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 2258.975231] Key type lgssc registered [ 2259.935380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2267.393300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2267.416398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.831900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.582725] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug +malloc all [ 2278.369153] LustreError: 75820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.381315] LustreError: 75820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2280.034785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.327113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.569398] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug +malloc all [ 2291.521866] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2297.027728] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2303.312919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2303.647900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.665837] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug +malloc all [ 2313.902486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2316.660404] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2324.450468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2324.451028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.460959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2324.475081] Lustre: Skipped 1 previous similar message [ 2329.572805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.581827] Lustre: Skipped 2 previous similar messages [ 2329.750614] Lustre: server umount lustre-OST0000 complete [ 2334.688072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.700429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.704657] Lustre: Skipped 1 previous similar message [ 2339.249911] Lustre: server umount lustre-MDT0000 complete [ 2339.814384] LustreError: 77061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.844039] LustreError: 77061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2343.081290] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615814 with bad export cookie 17020378660147868436 [ 2343.088894] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.101787] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2343.364267] Lustre: server umount lustre-MDT0001 complete [ 2354.202634] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 2356.825865] Key type lgssc unregistered [ 2357.114146] LNet: 79359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.120243] LNetError: 79359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.139279] LNet: Removed LNI 192.168.206.120@tcp [ 2357.964264] Key type .llcrypt unregistered [ 2357.966791] Key type ._llcrypt unregistered [ 2380.138138] Key type ._llcrypt registered [ 2380.140321] Key type .llcrypt registered [ 2380.245123] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2381.502569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2381.545348] alg: No test for adler32 (adler32-zlib) [ 2382.519245] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2382.740513] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 2384.455171] Key type lgssc registered [ 2385.543713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2393.906802] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2393.931966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2409.504767] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2414.623755] LustreError: 80748:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.120@tcp: failed processing log, type 4: rc = -110 [ 2441.375455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.625955] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2455.970909] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2463.631492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.750565] Lustre: *** cfs_fail_loc=706, val=0*** [ 2464.084423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.141971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2468.162938] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2469.344844] LustreError: 80774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.387071] LustreError: 80774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2474.465066] LustreError: 80773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.090881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.470816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2480.620497] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2487.709969] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2493.659911] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2496.998939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2497.004662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.018070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.020242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.020975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.044199] Lustre: Skipped 1 previous similar message [ 2503.143107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.153102] Lustre: Skipped 1 previous similar message [ 2508.256843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2510.303177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2510.434991] Lustre: server umount lustre-OST0000 complete [ 2517.473078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2517.507771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.524783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.527471] Lustre: Skipped 1 previous similar message [ 2528.223254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2528.458628] Lustre: server umount lustre-MDT0000 complete [ 2528.741483] LustreError: 81838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.775573] LustreError: 81838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2531.777834] LustreError: 80755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616002 with bad export cookie 6225584374301266390 [ 2531.784835] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2531.786671] LustreError: 80755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2532.063443] Lustre: server umount lustre-MDT0001 complete [ 2543.758702] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 2546.830739] Key type lgssc unregistered [ 2547.077272] LNet: 84369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2547.088050] LNetError: 84369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2547.116029] LNet: Removed LNI 192.168.206.120@tcp [ 2548.324535] Key type .llcrypt unregistered [ 2548.326625] Key type ._llcrypt unregistered [ 2571.957370] Key type ._llcrypt registered [ 2571.963601] Key type .llcrypt registered [ 2572.064411] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2572.870906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2572.915862] alg: No test for adler32 (adler32-zlib) [ 2573.926253] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2574.201404] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 2575.887157] Key type lgssc registered [ 2577.149473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2584.588368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2584.623756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.327692] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2596.309472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.767171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2600.384721] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2601.951835] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.963960] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2606.050786] LustreError: 86490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.075306] LustreError: 86490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2607.977282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2608.283517] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.296239] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2608.317613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.730468] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.752858] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2613.852039] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2618.850011] LustreError: 86489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.878024] LustreError: 86489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2621.263724] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.274404] Lustre: MGS: Client e0e25a7d-860a-48fc-a0c9-16f570756ab4 (at 0@lo) reconnecting [ 2621.311938] LustreError: 86500:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2621.524507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.617048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2625.531836] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2638.305536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.311473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.324708] Lustre: Skipped 1 previous similar message [ 2643.426134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2643.440996] Lustre: Skipped 2 previous similar messages [ 2644.492067] Lustre: server umount lustre-OST0000 complete [ 2648.546072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.553037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.567946] Lustre: Skipped 1 previous similar message [ 2648.578177] Lustre: Skipped 1 previous similar message [ 2653.666313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.674119] Lustre: Skipped 1 previous similar message [ 2658.784040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.369014] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2662.538051] Lustre: server umount lustre-MDT0000 complete [ 2663.905312] LustreError: 86489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.928977] LustreError: 86489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2666.463272] Lustre: server umount lustre-MDT0001 complete [ 2669.903772] Lustre: server umount MGS complete [ 2680.217943] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:02:30 (1787616150) [ 2707.519826] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 2715.267594] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2747.469123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.726770] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2761.254209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.472879] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2761.511384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2761.600919] Lustre: lustre-MDT0001: new disk, initializing [ 2761.715821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.752263] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2761.758526] Lustre: 93654:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2761.765538] LustreError: 93654:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2765.440835] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2776.686738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.989690] Lustre: lustre-OST0000: new disk, initializing [ 2777.005629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2777.013759] Lustre: 94420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2777.091728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.853754] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2783.843138] Lustre: 94761:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c4797e40 x1874451282041984/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787616270 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2791.337986] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2798.290228] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.304251] Lustre: MGS: Client f699fa68-3d05-4491-b7d4-7dae4b586522 (at 0@lo) reconnecting [ 2798.335893] Lustre: 92922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2798.357226] LustreError: 93640:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2798.365152] LustreError: 93640:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2798.381631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2798.463223] Lustre: lustre-MDT0000: new disk, initializing [ 2798.656064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.717409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2798.735306] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2799.787336] Lustre: 95488:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b43d619e x1874451282048512/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787616286 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2803.802593] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2804.849910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2804.876259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2804.965117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2830.309034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.316924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2830.318850] Lustre: Skipped 1 previous similar message [ 2830.326210] Lustre: Skipped 2 previous similar messages [ 2834.258525] Lustre: server umount lustre-OST0000 complete [ 2840.548554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.548827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2850.784892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.788129] Lustre: Skipped 5 previous similar messages [ 2852.319169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2852.422152] Lustre: server umount lustre-MDT0000 complete [ 2870.239213] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2870.535797] Lustre: server umount lustre-MDT0001 complete [ 2873.467694] Lustre: server umount MGS complete [ 2884.553887] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:05:54 (1787616354) [ 2904.951733] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 2908.737162] Key type lgssc unregistered [ 2909.299780] LNet: 97996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2909.329123] LNetError: 97996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2909.348333] LNet: Removed LNI 192.168.206.120@tcp [ 2910.656097] Key type .llcrypt unregistered [ 2910.657906] Key type ._llcrypt unregistered [ 2926.946422] Key type ._llcrypt registered [ 2926.949158] Key type .llcrypt registered [ 2927.074402] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2928.086898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2928.231140] alg: No test for adler32 (adler32-zlib) [ 2929.314165] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 2929.629913] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 2931.367164] Key type lgssc registered [ 2932.386847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.409853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2939.439391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.799925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.344508] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2950.113560] LustreError: 99220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.125409] LustreError: 99220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2952.421792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.854698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2956.914733] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2965.139877] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2971.275781] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2977.489583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2977.744750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.818472] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2985.978469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2991.959630] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3001.319054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.321402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3001.321803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3001.333411] Lustre: Skipped 1 previous similar message [ 3006.076869] Lustre: server umount lustre-OST0000 complete [ 3011.554716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.557055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.578785] Lustre: Skipped 1 previous similar message [ 3011.602253] Lustre: Skipped 2 previous similar messages [ 3015.906327] Lustre: server umount lustre-MDT0000 complete [ 3016.674689] LustreError: 99217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.707494] LustreError: 99217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3020.244335] LustreError: 99201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616491 with bad export cookie 3416213428424350256 [ 3020.256407] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.267155] LustreError: 99201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3020.822681] Lustre: server umount lustre-MDT0001 complete [ 3033.068178] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 3035.755907] Key type lgssc unregistered [ 3036.039751] LNet: 102752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.046818] LNetError: 102752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3036.064546] LNet: Removed LNI 192.168.206.120@tcp [ 3036.825716] Key type .llcrypt unregistered [ 3036.828357] Key type ._llcrypt unregistered [ 3054.202622] Key type ._llcrypt registered [ 3054.204234] Key type .llcrypt registered [ 3054.380937] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 3057.205884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3057.281186] alg: No test for adler32 (adler32-zlib) [ 3058.759849] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 3059.443638] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 3061.359274] Key type lgssc registered [ 3063.687803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3073.834246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3073.837531] LustreError: 103837:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3073.847165] Lustre: Skipped 1 previous similar message [ 3073.847184] LustreError: 103836:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3073.876244] LustreError: 103837:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3073.880660] LustreError: 103837:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3073.888631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.896157] LustreError: 103837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3075.461788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.799177] LustreError: 103875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.833976] LustreError: 103875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3086.820244] LustreError: 103874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.205129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.658287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3094.116690] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3099.904078] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing lsmod [ 3102.468060] LustreError: 105044:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3102.520787] LustreError: 105045:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3102.530625] LustreError: 105044:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4950 [ 3102.538238] LustreError: 105044:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3102.550616] LustreError: 105044:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3102.560558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.948716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3108.326729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.329155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.338986] Lustre: Skipped 1 previous similar message [ 3108.362963] Lustre: Skipped 2 previous similar messages [ 3111.738527] Lustre: server umount lustre-MDT0000 complete [ 3113.441741] LustreError: 103874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.464732] LustreError: 103874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3115.783519] LustreError: 103852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616586 with bad export cookie 4477214190019621901 [ 3115.789537] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.798768] LustreError: 103852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3116.643497] LustreError: 104307:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ee592c78 x1874451790583040/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3116.783409] Lustre: server umount lustre-MDT0001 complete [ 3131.627603] Lustre: server umount lustre-OST0000 complete [ 3141.618644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.173313] LustreError: 106109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.219085] LustreError: 106109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3142.303498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.739092] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3147.756029] LustreError: 106110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.612603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.040703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3160.563709] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3168.371136] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3174.224840] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3180.464392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.812610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.682275] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3187.968887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3195.784661] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3208.686417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3208.689665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.736379] Lustre: Skipped 1 previous similar message [ 3208.752794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.724354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.725385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3210.756140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.266972] Lustre: server umount lustre-OST0000 complete [ 3220.968481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.975040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3220.985880] Lustre: Skipped 1 previous similar message [ 3220.988158] Lustre: Skipped 1 previous similar message [ 3224.342888] Lustre: server umount lustre-MDT0000 complete [ 3226.080043] LustreError: 106109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.116994] LustreError: 106109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3227.991948] LustreError: 106091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616698 with bad export cookie 4477214190019622615 [ 3227.998811] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3228.001911] LustreError: 106091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3228.338757] Lustre: server umount lustre-MDT0001 complete [ 3239.189694] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 3242.383266] Key type lgssc unregistered [ 3242.713278] LNet: 109644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.719897] LNetError: 109644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.741563] LNet: Removed LNI 192.168.206.120@tcp [ 3243.708526] Key type .llcrypt unregistered [ 3243.714074] Key type ._llcrypt unregistered [ 3268.127716] Key type ._llcrypt registered [ 3268.130340] Key type .llcrypt registered [ 3268.292672] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 3269.454344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3269.570792] alg: No test for adler32 (adler32-zlib) [ 3271.075744] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 3271.478645] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 3273.271188] Key type lgssc registered [ 3274.996435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3284.129969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3284.159467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.596421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3290.695420] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3295.734246] LustreError: 111071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.761504] LustreError: 111071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3300.833940] LustreError: 111072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.781923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.227513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3307.562043] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3316.441286] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3322.409270] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3329.030366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.460762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.502345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3334.846366] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3341.482651] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3350.963252] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3354.566903] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3361.253101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.268745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.289733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.298890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.300594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.310080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.203976] Lustre: server umount lustre-OST0000 complete [ 3373.538785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.541164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.569426] Lustre: Skipped 1 previous similar message [ 3373.573182] Lustre: Skipped 1 previous similar message [ 3376.385528] Lustre: server umount lustre-MDT0000 complete [ 3378.656744] LustreError: 111071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.665908] LustreError: 111071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3379.965425] LustreError: 111052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616850 with bad export cookie 15040473515467099293 [ 3379.968687] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3379.979158] LustreError: 111052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3380.244493] Lustre: server umount lustre-MDT0001 complete [ 3391.484264] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 3394.426597] Key type lgssc unregistered [ 3394.708536] LNet: 114714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3394.714955] LNetError: 114714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3394.729495] LNet: Removed LNI 192.168.206.120@tcp [ 3395.640900] Key type .llcrypt unregistered [ 3395.642839] Key type ._llcrypt unregistered [ 3413.978390] Key type ._llcrypt registered [ 3413.981185] Key type .llcrypt registered [ 3414.085269] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 3415.232175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.245058] alg: No test for adler32 (adler32-zlib) [ 3416.259412] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 3416.571808] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 3418.257855] Key type lgssc registered [ 3419.581798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3427.217591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3427.248348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.643612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.261225] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3438.050579] LustreError: 115940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.074052] LustreError: 115940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3442.444561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.799331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.265718] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3456.343152] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3462.694715] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3470.651790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.194131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.285357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3477.680117] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3484.961185] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3498.994639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3499.011432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3499.038749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.072499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.075762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.087172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.104962] Lustre: Skipped 1 previous similar message [ 3504.353353] Lustre: server umount lustre-OST0000 complete [ 3513.318992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.328505] Lustre: Skipped 1 previous similar message [ 3513.330043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.341747] Lustre: Skipped 1 previous similar message [ 3516.196727] Lustre: server umount lustre-MDT0000 complete [ 3518.437598] LustreError: 115940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.461563] LustreError: 115940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3520.381056] LustreError: 115920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616991 with bad export cookie 16859841529496540011 [ 3520.385685] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.404399] LustreError: 115920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.790434] Lustre: server umount lustre-MDT0001 complete [ 3531.832856] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 3534.259680] Key type lgssc unregistered [ 3534.549160] LNet: 119494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3534.554250] LNetError: 119494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3534.563165] LNet: Removed LNI 192.168.206.120@tcp [ 3535.313693] Key type .llcrypt unregistered [ 3535.315592] Key type ._llcrypt unregistered [ 3556.495601] Key type ._llcrypt registered [ 3556.498575] Key type .llcrypt registered [ 3556.629528] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 3557.768488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3557.874497] alg: No test for adler32 (adler32-zlib) [ 3558.964239] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 3559.254227] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 3561.007670] Key type lgssc registered [ 3561.966379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3568.876458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3568.921969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.339578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.746101] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3580.904350] LustreError: 120888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.944522] LustreError: 120888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3585.122631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.578965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3590.486572] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3598.293927] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3604.538452] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3611.893057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3612.336340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.425377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3620.281755] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3629.017927] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3636.215222] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3639.892990] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3639.900308] Lustre: Skipped 2 previous similar messages [ 3643.463134] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3643.507266] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3644.000359] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3644.013412] Lustre: Skipped 2 previous similar messages [ 3653.727301] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3653.738825] Lustre: Skipped 1 previous similar message [ 3658.559713] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.20@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3658.583065] Lustre: Skipped 2 previous similar messages [ 3664.428953] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.20@tcp 192.168.0.[2,10]@tcp [ 3668.816461] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.20/24@tcp in log lustre-MDT0000 [ 3668.825165] Lustre: Skipped 2 previous similar messages [ 3671.663826] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.20/24@tcp [ 3671.677795] Lustre: Skipped 1 previous similar message [ 3682.281948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.284675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3682.299179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3682.300669] LustreError: Skipped 1 previous similar message [ 3682.309025] Lustre: Skipped 1 previous similar message [ 3686.608612] Lustre: server umount lustre-OST0000 complete [ 3692.538807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.561666] Lustre: Skipped 1 previous similar message [ 3692.573144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.588616] Lustre: Skipped 1 previous similar message [ 3697.632458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3697.644632] Lustre: Skipped 1 previous similar message [ 3702.760653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.312960] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3705.514152] Lustre: server umount lustre-MDT0000 complete [ 3707.873506] LustreError: 120888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.904911] LustreError: 120888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3709.902182] LustreError: 120871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617180 with bad export cookie 15118114240256930619 [ 3709.902841] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.918221] LustreError: 120871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3710.267093] Lustre: server umount lustre-MDT0001 complete [ 3721.469541] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 3724.459451] Key type lgssc unregistered [ 3724.759393] LNet: 125890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3724.770608] LNetError: 125890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3724.790660] LNet: Removed LNI 192.168.206.120@tcp [ 3725.646514] Key type .llcrypt unregistered [ 3725.648680] Key type ._llcrypt unregistered [ 3754.301964] Key type ._llcrypt registered [ 3754.303784] Key type .llcrypt registered [ 3754.435570] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 3755.542786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3755.579411] alg: No test for adler32 (adler32-zlib) [ 3756.684339] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 3756.920279] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 3758.719623] Key type lgssc registered [ 3759.823614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3768.229468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3768.263586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.786835] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3769.841113] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.20/24@tcp [ 3769.855703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.275275] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3779.045355] LustreError: 127452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.071057] LustreError: 127452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3784.166249] LustreError: 127453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.309427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.086558] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3786.130362] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.20/24@tcp [ 3786.196715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3792.559951] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3799.190403] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3805.823883] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3813.349531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3813.705521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.792307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3820.602979] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3828.394480] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3840.502697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3840.507425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.520723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3840.532086] Lustre: Skipped 1 previous similar message [ 3840.553218] Lustre: Skipped 1 previous similar message [ 3845.626722] LustreError: 129282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.771747] Lustre: server umount lustre-OST0000 complete [ 3850.726983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.739940] Lustre: Skipped 1 previous similar message [ 3850.740431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.749059] Lustre: Skipped 1 previous similar message [ 3855.844901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3855.855285] Lustre: Skipped 1 previous similar message [ 3856.377496] Lustre: server umount lustre-MDT0000 complete [ 3860.126306] LustreError: 127433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617331 with bad export cookie 7051064073283359562 [ 3860.131194] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.137510] LustreError: 127433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3860.554589] Lustre: server umount lustre-MDT0001 complete [ 3870.795558] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 3873.315726] Key type lgssc unregistered [ 3873.588353] LNet: 131045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3873.595355] LNetError: 131045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3873.612502] LNet: Removed LNI 192.168.206.120@tcp [ 3874.672995] Key type .llcrypt unregistered [ 3874.675944] Key type ._llcrypt unregistered [ 3901.416615] Key type ._llcrypt registered [ 3901.418727] Key type .llcrypt registered [ 3901.504645] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 3916.097915] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 3917.060749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3917.110141] alg: No test for adler32 (adler32-zlib) [ 3918.138502] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 3918.333898] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 3919.959213] Key type lgssc registered [ 3920.861322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3955.872651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3955.897571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.253688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3957.299139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3957.392319] Lustre: lustre-MDT0000: new disk, initializing [ 3957.470352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.480356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3962.363836] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3972.636358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.720795] Lustre: 134448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3972.795157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3972.804125] Lustre: Skipped 1 previous similar message [ 3972.897991] Lustre: lustre-MDT0001: new disk, initializing [ 3973.011726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3973.042910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3973.051309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3976.605288] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3986.858237] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3993.258295] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4001.779062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4002.269584] Lustre: lustre-OST0000: new disk, initializing [ 4002.274762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4002.280707] Lustre: 136415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4002.356468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.790798] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4008.503721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4008.514917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4008.633569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4016.992860] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4021.666617] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4021.839149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4026.690366] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4026.953287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4029.416178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4029.433208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.438957] LustreError: Skipped 1 previous similar message [ 4029.443690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4029.476981] Lustre: Skipped 1 previous similar message [ 4034.533734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4034.540043] Lustre: Skipped 1 previous similar message [ 4035.491143] Lustre: server umount lustre-OST0000 complete [ 4044.769506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.771080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4044.794259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4044.799951] Lustre: Skipped 2 previous similar messages [ 4046.120388] Lustre: server umount lustre-MDT0000 complete [ 4049.661661] LustreError: 136414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617520 with bad export cookie 15153334575074682845 [ 4049.667828] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4049.669687] LustreError: 136414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4049.930656] Lustre: server umount lustre-MDT0001 complete [ 4058.450629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.792832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.615525] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4064.224104] LustreError: 138630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.238675] LustreError: 138630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4068.325498] LustreError: 138631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.310749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.568493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.566466] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4082.727314] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4087.975134] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4094.550641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.067122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.958456] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4110.123856] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4114.093870] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4114.332938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4118.330883] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4118.567937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4127.225571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4127.479542] Lustre: lustre-OST0001: new disk, initializing [ 4127.486490] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4127.493710] Lustre: 142002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4127.638343] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4133.972705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4133.985692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4134.159858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4134.504995] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4143.900960] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4144.265498] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4149.571946] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4149.832692] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4155.890711] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98e308a22000.ost_server_uuid 50 [ 4157.564152] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98e308a22000.ost_server_uuid in IDLE state after 0 sec [ 4170.227974] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.236396] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.245332] Lustre: Skipped 1 previous similar message [ 4170.250546] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4174.696886] Lustre: server umount lustre-OST0001 complete [ 4175.354959] LustreError: 140454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.379185] LustreError: 140454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4180.452612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.455123] LustreError: 140454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.459529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4180.459537] Lustre: Skipped 1 previous similar message [ 4180.460062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4180.460067] LustreError: Skipped 1 previous similar message [ 4180.463772] Lustre: Skipped 1 previous similar message [ 4180.480351] LustreError: 140454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4185.577843] LustreError: 140451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4185.597632] LustreError: 140451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4190.697340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4190.707934] Lustre: Skipped 4 previous similar messages [ 4193.249897] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4193.446831] Lustre: server umount lustre-OST0000 complete [ 4199.394734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4199.404471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.167973] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4211.171252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4211.180150] Lustre: Skipped 6 previous similar messages [ 4211.337536] Lustre: server umount lustre-MDT0000 complete [ 4215.661664] LustreError: 138610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617686 with bad export cookie 15153334575074683727 [ 4215.666444] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.671374] LustreError: 138610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4216.022932] Lustre: server umount lustre-MDT0001 complete [ 4237.592702] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 4240.232640] Key type lgssc unregistered [ 4240.554659] LNet: 145012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4240.560129] LNetError: 145012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4240.578993] LNet: Removed LNI 192.168.206.120@tcp [ 4241.536786] Key type .llcrypt unregistered [ 4241.543280] Key type ._llcrypt unregistered [ 4278.189410] Key type ._llcrypt registered [ 4278.192320] Key type .llcrypt registered [ 4278.301719] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 4292.942339] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 4293.984519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4294.030792] alg: No test for adler32 (adler32-zlib) [ 4295.056584] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 4295.253965] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 4296.895239] Key type lgssc registered [ 4297.803743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4328.490905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4328.525223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.834492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4329.867811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4330.003926] Lustre: lustre-MDT0000: new disk, initializing [ 4330.101240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.117250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4335.570086] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4347.078328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.251171] Lustre: 149165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4347.333062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4347.338476] Lustre: Skipped 1 previous similar message [ 4347.456498] Lustre: lustre-MDT0001: new disk, initializing [ 4347.578573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4347.636566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4347.651555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4351.868218] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4360.138434] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4365.595430] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4372.113195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4372.332547] Lustre: lustre-OST0000: new disk, initializing [ 4372.336730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4372.344066] Lustre: 151133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4372.420395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.710263] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4382.835688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4382.847535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4382.946117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4388.290313] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4393.205579] Lustre: Failing over lustre-OST0000 [ 4393.453705] Lustre: server umount lustre-OST0000 complete [ 4409.231996] Lustre: 147392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787617864/real 1787617864] req@00000000ee8003f3 x1874453086633856/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787617880 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4409.273956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.810296] Lustre: 147394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787617864/real 1787617864] req@000000003ee81596 x1874453086633984/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787617880 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4409.849527] Lustre: 147394:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4409.872220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.361762] Lustre: 147392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787617866/real 1787617866] req@00000000e65aac05 x1874453086634240/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787617882 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4411.392332] Lustre: 147392:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4411.852746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4412.147455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.170869] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4415.210652] Lustre: 147394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787617869/real 1787617869] req@00000000039b33e3 x1874453086635136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787617885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4418.010470] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4419.314851] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4420.322711] Lustre: 147391:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787617874/real 1787617874] req@00000000936c2abd x1874453086636416/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787617890 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4420.367162] Lustre: 147391:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4421.833921] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4421.835647] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4422.400789] Lustre: Failing over lustre-MDT0000 [ 4422.851181] Lustre: server umount lustre-MDT0000 complete [ 4424.675121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4424.677921] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.705852] Lustre: Skipped 2 previous similar messages [ 4429.795317] LustreError: 150401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.816630] LustreError: 150401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4432.022135] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.913049] LustreError: 149173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.939039] LustreError: 149173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4437.134882] LustreError: 149172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.910805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.000809] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4441.200376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.234803] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4442.253583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4443.877346] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4446.700212] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4446.703211] Lustre: Skipped 1 previous similar message [ 4446.713676] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4446.752801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4451.813071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.822141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.824867] Lustre: Skipped 1 previous similar message [ 4451.835971] Lustre: Skipped 1 previous similar message [ 4456.339864] Lustre: server umount lustre-OST0000 complete [ 4462.049702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4462.055721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.095752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.168654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.192300] Lustre: Skipped 2 previous similar messages [ 4468.031453] Lustre: server umount lustre-MDT0000 complete [ 4471.284568] LustreError: 149177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.326341] LustreError: 149177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4472.681280] LustreError: 149159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617943 with bad export cookie 12022406672785935321 [ 4472.686650] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.700967] LustreError: 149159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4473.338734] Lustre: server umount lustre-MDT0001 complete [ 4482.756222] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 4485.057921] Key type lgssc unregistered [ 4485.262269] LNet: 154793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4485.275414] LNetError: 154793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4485.293433] LNet: Removed LNI 192.168.206.120@tcp [ 4485.930252] Key type .llcrypt unregistered [ 4485.932535] Key type ._llcrypt unregistered [ 4522.006674] Key type ._llcrypt registered [ 4522.009167] Key type .llcrypt registered [ 4522.117314] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 4522.971753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4522.984155] alg: No test for adler32 (adler32-zlib) [ 4523.968382] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 4524.171715] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 4525.800041] Key type lgssc registered [ 4526.648769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4533.178484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4533.202120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4534.540187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4538.063208] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4544.998488] LustreError: 156313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.017026] LustreError: 156313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4545.326671] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4548.855783] LustreError: 156342:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4548.864936] LustreError: 156342:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4548.870902] LustreError: 156342:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4549.137562] Lustre: server umount lustre-MDT0000 complete [ 4566.115079] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 4578.054337] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 4609.856586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.012270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4610.050635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4610.119712] Lustre: lustre-MDT0000: new disk, initializing [ 4610.189782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.201827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4613.878577] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4623.742216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.836793] Lustre: 159645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4623.887278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4623.891527] Lustre: Skipped 1 previous similar message [ 4623.957934] Lustre: lustre-MDT0001: new disk, initializing [ 4624.081099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.110871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4624.117767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4628.602902] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4637.885567] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4643.337258] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4651.040559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4651.370787] Lustre: lustre-OST0000: new disk, initializing [ 4651.378099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4651.388686] Lustre: 161612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4651.461920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.357842] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 4659.281560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4659.291237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4659.460302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4666.146789] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4669.547587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4670.869067] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:35:41 (1787618141) [ 4920.801797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.805662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.821875] Lustre: Skipped 2 previous similar messages [ 4920.831957] Lustre: Skipped 1 previous similar message [ 4925.807256] Lustre: server umount lustre-MDT0000 complete [ 4925.919728] LustreError: 161207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.936102] LustreError: 161207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4929.279691] LustreError: 161623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618400 with bad export cookie 15930306721939725343 [ 4929.282789] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4929.289511] LustreError: 161623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4931.039812] LustreError: 163382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.041626] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.064660] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4934.047825] LustreError: 159650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.048096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4935.569884] Lustre: server umount lustre-MDT0001 complete [ 4953.055151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4953.156405] Lustre: server umount lustre-OST0000 complete [ 4961.193946] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 4969.693993] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5005.355608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.692473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5005.742579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5005.855974] Lustre: lustre-MDT0000: new disk, initializing [ 5005.932638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.947722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5009.879699] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5020.343649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.465700] Lustre: 166277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5020.518443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5020.526356] Lustre: Skipped 1 previous similar message [ 5020.635477] Lustre: lustre-MDT0001: new disk, initializing [ 5020.816678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5020.876377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5020.902808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5025.714935] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5036.193570] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5042.198596] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5049.285885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5049.540769] Lustre: lustre-OST0000: new disk, initializing [ 5049.544043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5049.548406] Lustre: 168249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5049.641320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5055.048161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5055.061766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5055.204826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5056.013110] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5066.037382] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5071.132374] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5071.309941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5075.018962] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5075.186508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5081.058260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5081.059370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.070934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5081.073688] Lustre: Skipped 1 previous similar message [ 5081.080624] Lustre: Skipped 1 previous similar message [ 5086.182882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5086.187511] Lustre: Skipped 1 previous similar message [ 5091.295134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5091.417669] Lustre: server umount lustre-OST0000 complete [ 5096.416933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.420552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.433147] Lustre: Skipped 1 previous similar message [ 5096.446095] Lustre: Skipped 3 previous similar messages [ 5100.657699] Lustre: server umount lustre-MDT0000 complete [ 5101.536427] LustreError: 167505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.559711] LustreError: 167505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5103.739771] LustreError: 166270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618574 with bad export cookie 15930306721939884166 [ 5103.747281] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5103.756851] LustreError: 166270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5104.097539] Lustre: server umount lustre-MDT0001 complete [ 5113.739516] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:43:03 (1787618583) [ 5129.502685] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 5139.457259] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5176.103585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.299294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5176.320235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5176.383518] Lustre: lustre-MDT0000: new disk, initializing [ 5176.450791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5176.469845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5180.724798] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5190.265173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.383328] Lustre: 173006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5190.584480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5194.810218] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5203.307722] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5208.745971] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5214.661817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5215.038166] Lustre: lustre-OST0000: new disk, initializing [ 5215.044602] Lustre: Skipped 1 previous similar message [ 5215.050875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5215.055164] Lustre: Skipped 2 previous similar messages [ 5215.060082] Lustre: 174981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5221.062111] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5222.450753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5222.462366] Lustre: Skipped 1 previous similar message [ 5222.468467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5222.536075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5230.207783] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5238.244933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5238.248055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.250718] LustreError: Skipped 2 previous similar messages [ 5238.260884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5238.269535] Lustre: Skipped 1 previous similar message [ 5238.862591] Lustre: server umount lustre-OST0000 complete [ 5241.830853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5247.845763] Lustre: server umount lustre-MDT0000 complete [ 5248.480964] LustreError: 173011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.504298] LustreError: 173011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5250.848931] LustreError: 172996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618721 with bad export cookie 15930306721939885055 [ 5250.859582] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5250.859772] LustreError: 172996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5251.206722] Lustre: server umount lustre-MDT0001 complete [ 5258.457315] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:45:28 (1787618728) [ 5273.746743] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 5285.016788] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5320.612792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.872315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5320.898405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5320.972209] Lustre: lustre-MDT0000: new disk, initializing [ 5321.039776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5321.044290] Lustre: Skipped 2 previous similar messages [ 5321.057735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5324.752666] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5335.278859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5335.394501] Lustre: 179300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5335.643850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5339.341418] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5347.435640] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5352.225745] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5358.526292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.854361] Lustre: 181278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5364.413060] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5367.916707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5368.124968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5374.099536] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5383.654899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5383.661842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5383.671143] Lustre: Skipped 3 previous similar messages [ 5383.675673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5383.680637] Lustre: Skipped 4 previous similar messages [ 5388.352465] Lustre: server umount lustre-OST0000 complete [ 5399.714588] Lustre: server umount lustre-MDT0000 complete [ 5403.112128] LustreError: 179312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.126478] LustreError: 179312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5403.380736] LustreError: 179294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618874 with bad export cookie 15930306721939886140 [ 5403.381590] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5403.389565] LustreError: 179294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5403.730317] Lustre: server umount lustre-MDT0001 complete [ 5415.191327] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 5417.389283] Key type lgssc unregistered [ 5417.583901] LNet: 183138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5417.589749] LNetError: 183138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5417.602930] LNet: Removed LNI 192.168.206.120@tcp [ 5418.097283] Key type .llcrypt unregistered [ 5418.099039] Key type ._llcrypt unregistered [ 5448.774158] Key type ._llcrypt registered [ 5448.775694] Key type .llcrypt registered [ 5448.892468] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5450.368594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5450.393994] alg: No test for adler32 (adler32-zlib) [ 5451.542372] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 5451.750048] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 5453.560635] Key type lgssc registered [ 5454.874864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5462.039949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5462.074736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.540871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.022471] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5473.762321] LustreError: 184554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5473.798393] LustreError: 184554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5477.646464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5478.134531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5483.307633] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5490.771170] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5498.305350] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5505.929256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5506.174524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.211070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5511.623099] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5518.886908] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5544.421059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.422075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5544.436220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.444370] Lustre: Skipped 1 previous similar message [ 5545.895584] Lustre: server umount lustre-OST0000 complete [ 5549.551280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.563621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.576890] Lustre: Skipped 2 previous similar messages [ 5554.664825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5554.675624] Lustre: Skipped 2 previous similar messages [ 5555.660931] Lustre: server umount lustre-MDT0000 complete [ 5559.148317] LustreError: 186398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619030 with bad export cookie 9107386620689753416 [ 5559.156451] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5559.161451] LustreError: 186398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5559.449784] Lustre: server umount lustre-MDT0001 complete [ 5570.800285] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 5572.991236] Key type lgssc unregistered [ 5573.176769] LNet: 188115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5573.193795] LNetError: 188115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5573.231985] LNet: Removed LNI 192.168.206.120@tcp [ 5574.106260] Key type .llcrypt unregistered [ 5574.108341] Key type ._llcrypt unregistered [ 5601.616559] Key type ._llcrypt registered [ 5601.618695] Key type .llcrypt registered [ 5601.691572] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5603.149546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5603.194732] alg: No test for adler32 (adler32-zlib) [ 5604.426470] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 5604.575457] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 5606.240110] Key type lgssc registered [ 5607.665959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5615.618946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5615.635206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.965950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.269478] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5626.345519] LustreError: 189534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.361643] LustreError: 189534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5628.771718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.090697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5633.654640] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5641.491323] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.722201] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5653.076294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.325744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.133839] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5661.556227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5667.477939] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5674.977250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5674.988336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.002453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5675.688155] Lustre: lustre-OST0000: Not available for connect from 192.168.206.20@tcp (stopping) [ 5677.026119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5677.032624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.037304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5677.053799] Lustre: Skipped 1 previous similar message [ 5680.474642] Lustre: server umount lustre-OST0000 complete [ 5687.585956] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98e309283000.ost_server_uuid 50 [ 5689.448951] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5702.624616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.625784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5702.636976] Lustre: Skipped 1 previous similar message [ 5708.554312] Lustre: server umount lustre-MDT0000 complete [ 5708.770472] LustreError: 189530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.783223] LustreError: 189530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5712.417217] LustreError: 189513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619183 with bad export cookie 7652153772216694751 [ 5712.445207] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5712.832353] Lustre: server umount lustre-MDT0001 complete [ 5721.984132] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:53:12 (1787619192) [ 5729.199905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.508349] LustreError: 193452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.602156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.355789] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5734.881160] LustreError: 193453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.001611] LustreError: 193452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.008913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.600599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.927538] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5755.383840] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5760.689946] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5766.463951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5766.762726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.515944] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5774.964720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5782.634332] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5792.060883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5792.575806] Lustre: lustre-OST0001: new disk, initializing [ 5792.583967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5792.608102] Lustre: 196292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5792.792251] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5799.583153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5799.603535] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5799.835690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5801.498031] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5813.831285] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5819.363700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.369821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.379680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.388164] Lustre: Skipped 3 previous similar messages [ 5822.729174] Lustre: server umount lustre-OST0000 complete [ 5824.654392] LustreError: 195273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.672766] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5827.800458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5831.315801] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5831.496417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5840.883194] LustreError: 195272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5840.903722] LustreError: 195272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5851.104206] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5851.117307] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.119203] LustreError: Skipped 1 previous similar message [ 5851.121651] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5851.121658] Lustre: Skipped 3 previous similar messages [ 5851.142490] Lustre: Skipped 2 previous similar messages [ 5852.449432] Lustre: server umount lustre-OST0001 complete [ 5862.235914] Lustre: server umount lustre-MDT0000 complete [ 5865.183737] LustreError: 193433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619336 with bad export cookie 7652153772216695822 [ 5865.192129] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5865.211114] LustreError: 193433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5865.535406] Lustre: server umount lustre-MDT0001 complete [ 5884.424305] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:55:54 (1787619354) [ 5889.330926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.543794] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5889.569297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5889.747644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.807605] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5901.075077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.135018] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5901.159204] Lustre: 199730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5904.774378] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5910.691848] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5915.247420] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5922.302162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.544420] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5930.121786] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5932.757993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5938.979741] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5944.417140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.505935] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5949.873274] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5953.727561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5956.481978] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5959.150141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.172660] Lustre: Skipped 3 previous similar messages [ 5959.173016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.186608] Lustre: Skipped 6 previous similar messages [ 5965.338349] Lustre: server umount lustre-OST0000 complete [ 5967.502831] LustreError: 201580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.510976] LustreError: 201580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5979.618579] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5979.622038] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.340758] Lustre: server umount lustre-OST0001 complete [ 5988.320695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5988.329606] LustreError: Skipped 1 previous similar message [ 5994.178643] Lustre: server umount lustre-MDT0000 complete [ 5997.349572] LustreError: 202614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619468 with bad export cookie 7652153772216697229 [ 5997.350219] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.354133] LustreError: 202614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5997.570992] Lustre: server umount lustre-MDT0001 complete [ 6020.442955] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:58:10 (1787619490) [ 6035.039884] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 6042.491346] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6070.343616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.532405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6070.564904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6070.659797] Lustre: lustre-MDT0000: new disk, initializing [ 6070.725106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.730251] Lustre: Skipped 3 previous similar messages [ 6070.742725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6074.885149] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6083.583127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.697836] Lustre: 207820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6083.785149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6083.792171] Lustre: Skipped 1 previous similar message [ 6083.886610] Lustre: lustre-MDT0001: new disk, initializing [ 6084.039890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6084.051894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6088.306944] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6096.496305] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6101.862869] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6107.030889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6107.206286] Lustre: lustre-OST0000: new disk, initializing [ 6107.209242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6107.213560] Lustre: 209788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6113.075362] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6115.428471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6115.439830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6115.584027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6121.620864] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6126.407979] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6126.598637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6130.951295] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6131.172651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6136.289502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.290940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.296308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.296315] Lustre: Skipped 9 previous similar messages [ 6136.318216] Lustre: Skipped 4 previous similar messages [ 6138.862345] Lustre: server umount lustre-OST0000 complete [ 6147.680092] Lustre: server umount lustre-MDT0000 complete [ 6150.826945] LustreError: 207813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619621 with bad export cookie 7652153772216699007 [ 6150.827663] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6150.834283] LustreError: 207813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6151.104079] Lustre: server umount lustre-MDT0001 complete [ 6157.896306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.413125] LustreError: 212003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.439047] LustreError: 212003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6163.106520] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6172.146679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.329869] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6181.419035] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6185.383872] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6190.726498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.026837] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6206.433229] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6211.330306] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6211.554098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6216.037056] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6216.175982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6217.697682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.702580] LustreError: Skipped 1 previous similar message [ 6217.706454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.714919] Lustre: Skipped 2 previous similar messages [ 6217.718462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.722433] Lustre: Skipped 5 previous similar messages [ 6223.839997] Lustre: server umount lustre-OST0000 complete [ 6229.866525] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6234.189288] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6262.845520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6263.290301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6263.296966] Lustre: Skipped 5 previous similar messages [ 6269.292933] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6269.425198] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.442078] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6277.402364] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6282.471973] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6284.836093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.854417] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6289.891315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6289.905281] LustreError: Skipped 1 previous similar message [ 6301.152103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6301.283025] Lustre: server umount lustre-OST0000 complete [ 6310.583092] Lustre: server umount lustre-MDT0000 complete [ 6314.754221] LustreError: 211984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619785 with bad export cookie 7652153772216699896 [ 6314.760158] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6314.771028] LustreError: 211984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6315.618790] Lustre: server umount lustre-MDT0001 complete [ 6326.057597] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:03:15 (1787619795) [ 6336.549358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6337.592951] LustreError: 217986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6337.625925] LustreError: 217986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6344.655996] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6355.662357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.007595] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6367.150634] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6374.418486] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6383.733206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.043196] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6398.177429] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6403.958887] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6408.523889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6414.714591] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6414.965106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6422.647565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.886215] Lustre: lustre-OST0001: new disk, initializing [ 6422.889530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6422.896210] Lustre: 221389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6428.811685] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6432.294964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6432.303377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6432.413893] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6436.923953] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6441.733645] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6442.000657] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6446.518265] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6446.766632] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6453.224098] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6453.238650] LustreError: Skipped 1 previous similar message [ 6453.249304] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.269410] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6453.280248] Lustre: Skipped 6 previous similar messages [ 6453.304409] Lustre: Skipped 13 previous similar messages [ 6455.338459] Lustre: server umount lustre-OST0001 complete [ 6462.612867] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6462.887354] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6468.024278] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6468.293830] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6496.975627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.375879] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6502.384848] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6502.385352] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6502.422853] Lustre: Skipped 1 previous similar message [ 6503.039612] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6512.761825] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6515.186426] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6524.200164] Lustre: server umount lustre-OST0001 complete [ 6546.638160] Lustre: server umount lustre-OST0000 complete [ 6556.475543] Lustre: server umount lustre-MDT0000 complete [ 6559.166401] LustreError: 217968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620030 with bad export cookie 7652153772216700862 [ 6559.169494] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6559.173445] LustreError: 217968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6565.715359] Lustre: server umount lustre-MDT0001 complete [ 6598.798307] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:07:48 (1787620068) [ 6605.873460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.158703] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6606.213323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6606.494231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6606.505937] Lustre: Skipped 5 previous similar messages [ 6611.022385] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6619.246605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.342071] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6619.380378] Lustre: 226633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6623.046903] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6630.385453] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6636.518173] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6644.244635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6644.699115] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6650.969115] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6660.443512] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6668.373578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6668.466086] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6676.899076] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6685.651349] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6688.052848] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6692.752768] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6692.942328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6696.708217] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6696.871364] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6702.172274] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98e30a2dd000.ost_server_uuid 50 [ 6704.139577] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98e30a2dd000.ost_server_uuid in IDLE state after 0 sec [ 6705.042433] Lustre: Permanently deactivating lustre-OST0001 [ 6705.053631] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6707.905475] Lustre: Permanently reactivating lustre-OST0001 [ 6712.801718] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6712.804269] LustreError: Skipped 5 previous similar messages [ 6712.805597] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.808311] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6712.827921] Lustre: Skipped 7 previous similar messages [ 6712.834372] Lustre: Skipped 8 previous similar messages [ 6717.869791] Lustre: server umount lustre-OST0001 complete [ 6717.925387] LustreError: 228481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.933806] LustreError: 228481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 6727.773837] Lustre: server umount lustre-OST0000 complete [ 6738.801557] Lustre: server umount lustre-MDT0000 complete [ 6742.781280] LustreError: 229507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620213 with bad export cookie 7652153772216702150 [ 6742.788068] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6742.794887] LustreError: 229507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6743.128428] Lustre: server umount lustre-MDT0001 complete [ 6774.667758] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:10:44 (1787620244) [ 6782.122030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.455502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6782.503287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6782.516145] Lustre: Skipped 5 previous similar messages [ 6785.945745] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6794.955954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.060358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6795.078680] Lustre: 233326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6799.712061] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6810.142334] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6817.819713] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6827.975626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.230387] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6828.247788] Lustre: Permanently deactivating lustre-OST0000 [ 6828.254769] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6837.892834] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6838.342138] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6848.533288] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6855.531741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6855.603708] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6861.402100] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6862.083263] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6870.440837] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6872.678129] Lustre: Permanently reactivating lustre-OST0000 [ 6872.694378] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6872.700510] Lustre: Skipped 2 previous similar messages [ 6882.967709] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6882.993484] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 00000000aad9ddf9, cur 1787620353 deadline 1787620339 last 1787620309 [ 6883.111067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6895.372406] Lustre: server umount lustre-OST0001 complete [ 6904.805478] Lustre: server umount lustre-OST0000 complete [ 6914.311229] Lustre: server umount lustre-MDT0000 complete [ 6918.089875] LustreError: 233320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620389 with bad export cookie 7652153772216703830 [ 6918.095595] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6918.118634] LustreError: 233320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6918.388152] Lustre: server umount lustre-MDT0001 complete [ 6928.622278] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 6932.471875] Key type lgssc unregistered [ 6932.811661] LNet: 238214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6932.825840] LNetError: 238214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6932.863085] LNet: Removed LNI 192.168.206.120@tcp [ 6933.830961] Key type .llcrypt unregistered [ 6933.832984] Key type ._llcrypt unregistered [ 6958.096482] Key type ._llcrypt registered [ 6958.100049] Key type .llcrypt registered [ 6958.309711] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6959.201601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6959.311282] alg: No test for adler32 (adler32-zlib) [ 6960.476789] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 6960.974296] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 6962.799174] Key type lgssc registered [ 6964.286568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6975.641970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6975.659444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.997644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6981.036834] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6986.211472] LustreError: 239684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.275851] LustreError: 239684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6991.012599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6991.224466] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6991.315653] Lustre: 239674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6991.335689] LustreError: 239683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.346844] Lustre: Permanently deactivating lustre-MDT0001 [ 6991.386402] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6991.762099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.236872] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6992.241240] LustreError: 239714:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6992.248821] LustreError: 239714:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6992.255232] LustreError: 239714:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6996.894876] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7006.595382] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7012.400729] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7021.114954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7021.556816] LustreError: 241525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.665662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7026.661265] LustreError: 241526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.700607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7028.658435] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7031.782091] LustreError: 241524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.629706] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7042.019303] LustreError: 241526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.035283] LustreError: 241526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7043.617627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.902363] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7049.218264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7049.222939] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7049.663060] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7058.242418] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.294643] Lustre: Permanently reactivating lustre-MDT0001 [ 7061.320317] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7061.339525] Lustre: Skipped 2 previous similar messages [ 7066.334831] Lustre: lustre-MDT0001: Client 3393ca3e-901f-4b07-bba3-b6072bdc9422 (at 192.168.206.20@tcp) reconnecting [ 7073.873531] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7076.202353] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7082.884200] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7083.124931] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7087.979334] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7088.316784] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7091.095744] Lustre: Permanently deactivating lustre-MDT0001 [ 7091.126468] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7091.142964] Lustre: Skipped 2 previous similar messages [ 7101.483221] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7114.288735] Lustre: server umount lustre-MDT0000 complete [ 7114.721332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7114.731258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.751195] LustreError: 243873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.760360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.779112] Lustre: Skipped 2 previous similar messages [ 7117.896900] LustreError: 239664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620588 with bad export cookie 1175682225101883783 [ 7117.903551] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7117.912514] LustreError: 239664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7118.188342] Lustre: server umount lustre-MDT0001 complete [ 7127.835917] Lustre: server umount lustre-OST0000 complete [ 7141.991350] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:16:51 (1787620611) [ 7153.296399] Lustre: server umount lustre-OST0001 complete [ 7160.742419] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 7172.942773] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7211.218459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.608319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7211.636070] Lustre: Skipped 2 previous similar messages [ 7211.686520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7211.752917] Lustre: lustre-MDT0000: new disk, initializing [ 7211.894578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7211.925106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7216.582647] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7226.369936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.459634] Lustre: 247992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7226.504698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7226.509145] Lustre: Skipped 1 previous similar message [ 7226.588721] Lustre: lustre-MDT0001: new disk, initializing [ 7226.680280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7226.715859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7226.724054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7231.078666] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7241.459559] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7247.874379] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7255.995687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.359467] Lustre: lustre-OST0000: new disk, initializing [ 7256.365334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7256.375992] Lustre: 249959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7256.430338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7261.311177] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7261.727885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7261.746699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7261.806757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7269.339436] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.169049] LustreError: 249086:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7279.496918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7279.793713] Lustre: lustre-OST0001: new disk, initializing [ 7279.798661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7279.805350] Lustre: 251072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7281.215098] LustreError: 249086:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7281.254137] LustreError: 247996:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7285.836754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7285.850474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7285.872769] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7285.969669] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7291.320076] LustreError: 247996:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7295.173107] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7296.880820] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7299.243402] Lustre: lustre-OST0001: Not available for connect from 192.168.206.20@tcp (stopping) [ 7301.601792] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.610890] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.624548] LustreError: Skipped 1 previous similar message [ 7301.627521] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7301.633187] Lustre: Skipped 1 previous similar message [ 7301.649197] Lustre: Skipped 1 previous similar message [ 7304.332243] Lustre: lustre-OST0001: Not available for connect from 192.168.206.20@tcp (stopping) [ 7304.704996] Lustre: server umount lustre-OST0001 complete [ 7306.731941] LustreError: 249943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.750382] LustreError: 249943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7322.091372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7322.107280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.206496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.222829] Lustre: Skipped 2 previous similar messages [ 7327.548021] Lustre: server umount lustre-OST0000 complete [ 7332.323569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.339902] Lustre: Skipped 2 previous similar messages [ 7337.442804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7337.447159] Lustre: Skipped 3 previous similar messages [ 7337.683212] Lustre: server umount lustre-MDT0000 complete [ 7341.570348] LustreError: 249957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620812 with bad export cookie 1175682225101886380 [ 7341.580667] LustreError: 249957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7341.581267] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.992791] Lustre: server umount lustre-MDT0001 complete [ 7357.707824] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 7360.093699] Key type lgssc unregistered [ 7360.457937] LNet: 253110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7360.465052] LNetError: 253110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7360.482956] LNet: Removed LNI 192.168.206.120@tcp [ 7361.502605] Key type .llcrypt unregistered [ 7361.504359] Key type ._llcrypt unregistered [ 7404.372290] Key type ._llcrypt registered [ 7404.377836] Key type .llcrypt registered [ 7404.513210] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7405.741067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7405.860699] alg: No test for adler32 (adler32-zlib) [ 7407.018220] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 7407.383064] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 7409.111466] Key type lgssc registered [ 7410.507097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7418.193799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7418.210869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.517791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7419.531279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7419.653492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7424.605312] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7435.404956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.586396] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7435.627929] Lustre: 255319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7436.089719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7441.100516] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7448.209851] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7453.139903] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7460.167419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7460.311143] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7460.513826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.426401] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7469.836990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7477.097935] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7536.608895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7536.617364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7536.631550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7541.116566] Lustre: server umount lustre-OST0000 complete [ 7545.819666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7561.603940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.068513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7568.700505] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7576.334170] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7576.431861] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7576.453093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7576.453621] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7576.484896] Lustre: Skipped 1 previous similar message [ 7617.506605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7617.510359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7617.518295] LustreError: Skipped 2 previous similar messages [ 7617.520842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7617.520849] Lustre: Skipped 1 previous similar message [ 7617.539439] Lustre: Skipped 2 previous similar messages [ 7621.028443] Lustre: server umount lustre-OST0000 complete [ 7625.697815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7625.718077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.742154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7625.750522] Lustre: Skipped 1 previous similar message [ 7627.744525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.749578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.785623] Lustre: Skipped 1 previous similar message [ 7630.655592] Lustre: server umount lustre-MDT0000 complete [ 7634.389223] LustreError: 255312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621105 with bad export cookie 630612872107994192 [ 7634.396575] LustreError: 255312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7634.401198] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.773475] Lustre: server umount lustre-MDT0001 complete [ 7645.129880] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 7647.900335] Key type lgssc unregistered [ 7648.179837] LNet: 260749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7648.185971] LNetError: 260749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7648.197489] LNet: Removed LNI 192.168.206.120@tcp [ 7649.053203] Key type .llcrypt unregistered [ 7649.054804] Key type ._llcrypt unregistered [ 7672.953583] Key type ._llcrypt registered [ 7672.957564] Key type .llcrypt registered [ 7673.057694] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7674.233344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7674.375094] alg: No test for adler32 (adler32-zlib) [ 7675.496471] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 7675.723181] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 7677.384416] Key type lgssc registered [ 7678.466773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7686.162365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7686.195623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.670063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7691.849574] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7697.890805] LustreError: 262170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.917748] LustreError: 262170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7701.176904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.658251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7706.439750] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7715.125451] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7721.828761] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7729.426504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7729.965047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.056920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7737.200354] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7745.388892] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7769.060941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7769.076646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.100626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7772.128851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7772.129782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7772.145458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.455770] Lustre: server umount lustre-OST0000 complete [ 7778.274872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7778.286515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.300594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.306241] Lustre: Skipped 1 previous similar message [ 7782.381346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7782.386373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7784.465536] Lustre: server umount lustre-MDT0000 complete [ 7787.492945] LustreError: 264152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7787.509921] LustreError: 264152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7787.932073] LustreError: 262150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621258 with bad export cookie 7471743205094704314 [ 7787.937305] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.949358] LustreError: 262150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7788.336878] Lustre: server umount lustre-MDT0001 complete [ 7798.836602] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 7801.478077] Key type lgssc unregistered [ 7801.769960] LNet: 266480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7801.777729] LNetError: 266480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7801.798070] LNet: Removed LNI 192.168.206.120@tcp [ 7802.543383] Key type .llcrypt unregistered [ 7802.545928] Key type ._llcrypt unregistered [ 7818.166138] Key type ._llcrypt registered [ 7818.167971] Key type .llcrypt registered [ 7818.255110] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7819.508541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7819.569784] alg: No test for adler32 (adler32-zlib) [ 7820.675928] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 7820.964232] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 7822.735251] Key type lgssc registered [ 7823.909641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7831.164426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7831.179765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.674236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.370957] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7842.786321] LustreError: 267699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7842.795772] LustreError: 267699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7844.542547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.856872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7850.029853] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7858.481928] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7864.286703] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7871.266657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7871.744652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7877.830282] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7877.866654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7884.953219] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7898.605981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7898.622071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.644492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7899.618932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7899.623538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7899.629565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.522255] Lustre: server umount lustre-OST0000 complete [ 7909.859220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.877525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7909.880383] Lustre: Skipped 2 previous similar messages [ 7914.605703] Lustre: server umount lustre-MDT0000 complete [ 7914.977472] LustreError: 267699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7915.007027] LustreError: 267699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7919.025869] LustreError: 267679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621389 with bad export cookie 4869668480743874071 [ 7919.051163] LustreError: 267679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7919.053465] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7919.603942] Lustre: server umount lustre-MDT0001 complete [ 7931.327684] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 7933.698498] Key type lgssc unregistered [ 7933.948455] LNet: 271377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7933.960673] LNetError: 271377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7933.971695] LNet: Removed LNI 192.168.206.120@tcp [ 7934.714617] Key type .llcrypt unregistered [ 7934.717410] Key type ._llcrypt unregistered [ 7958.635710] Key type ._llcrypt registered [ 7958.637546] Key type .llcrypt registered [ 7958.752452] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7960.166940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7960.409126] alg: No test for adler32 (adler32-zlib) [ 7961.617457] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 7961.812652] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 7963.439173] Key type lgssc registered [ 7964.718755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7972.985855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7973.025331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.524470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.347560] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7984.101584] LustreError: 272794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7984.120216] LustreError: 272794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7986.467559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.959094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7990.335712] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7997.525708] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8002.849634] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8010.654202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8011.002706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.204282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8019.251492] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8028.907489] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8050.146661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8050.154948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8050.193404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8055.267827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8055.277337] Lustre: Skipped 1 previous similar message [ 8056.112832] Lustre: server umount lustre-OST0000 complete [ 8060.385765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8060.414205] Lustre: Skipped 1 previous similar message [ 8060.424862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8060.443527] Lustre: Skipped 1 previous similar message [ 8065.511501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8065.517080] Lustre: Skipped 1 previous similar message [ 8065.857547] Lustre: server umount lustre-MDT0000 complete [ 8069.512174] LustreError: 272776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621540 with bad export cookie 18241467527356619006 [ 8069.520331] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8069.530832] LustreError: 272776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8069.874166] Lustre: server umount lustre-MDT0001 complete [ 8082.756727] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 8085.537407] Key type lgssc unregistered [ 8085.997734] LNet: 277103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8086.005404] LNetError: 277103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.027305] LNet: Removed LNI 192.168.206.120@tcp [ 8086.975977] Key type .llcrypt unregistered [ 8086.978219] Key type ._llcrypt unregistered [ 8108.376319] Key type ._llcrypt registered [ 8108.384716] Key type .llcrypt registered [ 8108.507389] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8110.268095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8110.347828] alg: No test for adler32 (adler32-zlib) [ 8111.620625] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 8112.052245] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 8113.711203] Key type lgssc registered [ 8115.099781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8124.398094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8124.448102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.104635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8131.187507] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8136.686805] LustreError: 278342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8136.740674] LustreError: 278342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8141.078083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.707043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8147.146266] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8155.782076] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8161.776715] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8169.364514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.728214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.821664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8175.853744] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8184.358922] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8197.613658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8197.621724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.631632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.705024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.710795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8201.731160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8202.927337] Lustre: server umount lustre-OST0000 complete [ 8208.352932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8208.357961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.367870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8208.370913] Lustre: Skipped 1 previous similar message [ 8211.937339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8211.943639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.354076] Lustre: server umount lustre-MDT0000 complete [ 8217.056542] LustreError: 278341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8217.086757] LustreError: 278341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8217.411690] LustreError: 281293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621688 with bad export cookie 11774307909440849879 [ 8217.416291] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8217.423245] LustreError: 281293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8217.939480] Lustre: server umount lustre-MDT0001 complete [ 8230.026763] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 8232.940728] Key type lgssc unregistered [ 8233.229780] LNet: 282049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8233.238768] LNetError: 282049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8233.258709] LNet: Removed LNI 192.168.206.120@tcp [ 8234.360646] Key type .llcrypt unregistered [ 8234.364791] Key type ._llcrypt unregistered [ 8246.414078] Key type ._llcrypt registered [ 8246.420131] Key type .llcrypt registered [ 8246.574201] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8265.187471] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 8280.536056] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8281.811138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8281.883791] alg: No test for adler32 (adler32-zlib) [ 8283.089562] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 8283.390295] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 8285.079187] Key type lgssc registered [ 8286.118621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8323.738418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8323.782229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.021450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8325.070469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8325.198132] Lustre: lustre-MDT0000: new disk, initializing [ 8325.286655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8325.302178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8329.808533] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8340.556561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.670241] Lustre: 285608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8340.738806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8340.748777] Lustre: Skipped 1 previous similar message [ 8340.846409] Lustre: lustre-MDT0001: new disk, initializing [ 8340.952358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8340.995188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8341.013200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8346.300129] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8356.066904] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8362.900726] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8370.357264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8370.630243] Lustre: lustre-OST0000: new disk, initializing [ 8370.633890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8370.645807] Lustre: 287580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8370.736106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.896837] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8381.033349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8381.059781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8381.162752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8388.816993] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8393.809511] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8393.990911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8399.142603] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8399.399214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8401.888550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8401.896818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.907563] LustreError: Skipped 1 previous similar message [ 8401.909230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8401.944934] Lustre: Skipped 1 previous similar message [ 8407.015870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8407.027786] Lustre: Skipped 1 previous similar message [ 8407.672824] Lustre: server umount lustre-OST0000 complete [ 8412.128141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8412.137804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.148557] Lustre: Skipped 1 previous similar message [ 8412.164756] Lustre: Skipped 2 previous similar messages [ 8417.248901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8417.263877] Lustre: Skipped 1 previous similar message [ 8418.628298] Lustre: server umount lustre-MDT0000 complete [ 8422.981453] LustreError: 286405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621893 with bad export cookie 12413732817154521535 [ 8422.993147] LustreError: 286405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8422.993275] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8423.305748] Lustre: server umount lustre-MDT0001 complete [ 8433.671573] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:38:23 (1787621903) [ 8441.374597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.843035] LustreError: 289990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8441.864950] LustreError: 289990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8441.937702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8446.658474] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8446.947078] LustreError: 289991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8451.040682] LustreError: 289990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.161043] LustreError: 289991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.352059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8456.751139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8461.934241] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8470.076810] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8476.773909] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8484.079434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8484.516113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8491.226609] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8499.853526] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8512.480199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8512.487624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8512.490568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8512.508211] Lustre: Skipped 1 previous similar message [ 8512.518666] Lustre: Skipped 1 previous similar message [ 8512.806551] Lustre: server umount lustre-OST0000 complete [ 8517.607796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.021301] Lustre: server umount lustre-MDT0000 complete [ 8522.739471] LustreError: 289985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8522.767320] LustreError: 289985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8523.176529] LustreError: 289971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621994 with bad export cookie 12413732817154522417 [ 8523.181268] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8523.182371] LustreError: 289971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8523.442886] Lustre: server umount lustre-MDT0001 complete [ 8537.772363] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 8540.975459] Key type lgssc unregistered [ 8541.300956] LNet: 293530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8541.309931] LNetError: 293530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8541.351551] LNet: Removed LNI 192.168.206.120@tcp [ 8542.289270] Key type .llcrypt unregistered [ 8542.291276] Key type ._llcrypt unregistered [ 8569.767538] Key type ._llcrypt registered [ 8569.771967] Key type .llcrypt registered [ 8569.940040] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 8583.635666] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8584.610746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8584.710052] alg: No test for adler32 (adler32-zlib) [ 8585.793622] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 8586.047487] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 8587.759471] Key type lgssc registered [ 8588.805668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8630.757839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8630.791846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.978963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8631.999271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8632.101778] Lustre: lustre-MDT0000: new disk, initializing [ 8632.204090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.236720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8636.761683] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8648.597134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.855532] Lustre: 297351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8648.946025] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8648.956830] Lustre: Skipped 1 previous similar message [ 8649.104275] Lustre: lustre-MDT0001: new disk, initializing [ 8649.278326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8649.344244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8649.361380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8655.412858] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8666.092537] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8672.133799] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8679.169979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8679.542614] Lustre: lustre-OST2710: new disk, initializing [ 8679.547569] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8679.556706] Lustre: 299320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8679.653628] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8686.817933] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8688.741936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8688.752395] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8688.891620] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8697.778200] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8705.660169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8705.798789] Lustre: lustre-OST03e8: new disk, initializing [ 8705.801677] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8705.806375] Lustre: 300385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8705.870344] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8711.755863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8711.767223] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8711.879743] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8712.019570] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8720.119593] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8728.729189] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8729.025840] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8734.278734] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8734.565542] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8739.841772] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8740.059968] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8744.581931] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8744.767151] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8750.659242] Lustre: server umount lustre-MDT0000 complete [ 8751.592150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8751.608220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8753.120297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.513573] LustreError: 299319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622225 with bad export cookie 14679061282506171830 [ 8754.525253] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8754.531359] LustreError: 299319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8754.896749] Lustre: server umount lustre-MDT0001 complete [ 8768.349641] Lustre: server umount lustre-OST2710 complete [ 8782.453976] Lustre: server umount lustre-OST03e8 complete [ 8806.088227] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 8820.789316] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8860.854449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.106462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8861.146494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8861.282675] Lustre: lustre-MDT0000: new disk, initializing [ 8861.512904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.542295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8869.414674] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8881.495357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.620428] Lustre: 305648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8881.722641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8881.732167] Lustre: Skipped 1 previous similar message [ 8881.791416] Lustre: lustre-MDT0001: new disk, initializing [ 8881.904331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8881.941399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8881.945808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8886.063220] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8898.052735] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8904.624882] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8912.669447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8913.073212] Lustre: lustre-OST0000: new disk, initializing [ 8913.082344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8913.091707] Lustre: 307619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8913.269584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8920.687041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8920.696425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8920.860700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8921.227626] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8931.012122] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8935.899780] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8936.121703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8941.470911] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8941.786214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8946.660491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.668213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8946.668820] LustreError: Skipped 1 previous similar message [ 8946.669397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.712686] Lustre: Skipped 3 previous similar messages [ 8950.344456] Lustre: server umount lustre-OST0000 complete [ 8956.903403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.905405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8956.926574] Lustre: Skipped 1 previous similar message [ 8956.939848] Lustre: Skipped 2 previous similar messages [ 8961.308973] Lustre: server umount lustre-MDT0000 complete [ 8962.017591] LustreError: 305658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8962.042073] LustreError: 305658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8965.310190] LustreError: 305639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622436 with bad export cookie 14679061282506173517 [ 8965.327543] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.335650] LustreError: 305639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8965.818562] Lustre: server umount lustre-MDT0001 complete [ 8976.803034] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:47:26 (1787622446) [ 8978.365276] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8980.519710] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:47:30 (1787622450) [ 8985.129735] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 9015.039652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.222708] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9015.237619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9015.427045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9020.168943] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9029.052403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.171924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9029.227422] Lustre: 311476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9034.551163] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9055.813505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9056.006669] Lustre: Denying initial registration attempt from nid 192.168.206.120@tcp, specified as failover [ 9056.014894] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9056.021391] LustreError: 312929:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9056.028906] LustreError: 312929:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9056.032462] LustreError: 312929:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9056.132814] Lustre: server umount lustre-OST0000 complete [ 9056.138417] LustreError: 312929:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9065.442895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9065.456048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9065.468673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9069.817260] Lustre: server umount lustre-MDT0000 complete [ 9072.097483] LustreError: 311488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9072.106529] LustreError: 311488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9073.589663] LustreError: 311468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622544 with bad export cookie 14679061282506174399 [ 9073.596384] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9073.609079] LustreError: 311468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9073.867040] Lustre: server umount lustre-MDT0001 complete [ 9086.274540] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9089.198623] Key type lgssc unregistered [ 9089.465774] LNet: 314050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9089.480197] LNetError: 314050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9089.503285] LNet: Removed LNI 192.168.206.120@tcp [ 9090.433198] Key type .llcrypt unregistered [ 9090.437524] Key type ._llcrypt unregistered [ 9104.704724] Key type ._llcrypt registered [ 9104.706463] Key type .llcrypt registered [ 9104.795502] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 9105.794125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9105.838780] alg: No test for adler32 (adler32-zlib) [ 9107.072506] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 9107.261805] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 9108.991166] Key type lgssc registered [ 9109.853609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9157.084630] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 9164.422334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9164.462311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.694800] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9165.708814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9165.870685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.825928] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9178.371206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.469519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9178.502300] Lustre: 316820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9178.755835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9183.213318] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9189.515360] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9196.411608] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9218.480850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9218.837961] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9219.081768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.580216] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9235.515449] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9240.032470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9240.042424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.081324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9240.550203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.654864] Lustre: server umount lustre-OST0000 complete [ 9250.785042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9250.785504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.789865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9250.799995] Lustre: Skipped 1 previous similar message [ 9250.802923] Lustre: Skipped 1 previous similar message [ 9254.660922] Lustre: server umount lustre-MDT0000 complete [ 9260.323372] LustreError: 316812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622731 with bad export cookie 2170760943368302743 [ 9260.326270] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9260.394341] LustreError: 316812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9260.890594] Lustre: server umount lustre-MDT0001 complete [ 9273.097077] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9275.811536] Key type lgssc unregistered [ 9276.079021] LNet: 320397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9276.090718] LNetError: 320397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9276.116552] LNet: Removed LNI 192.168.206.120@tcp [ 9277.233400] Key type .llcrypt unregistered [ 9277.236466] Key type ._llcrypt unregistered [ 9304.894641] Key type ._llcrypt registered [ 9304.901791] Key type .llcrypt registered [ 9305.060585] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 9306.441727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9306.456124] alg: No test for adler32 (adler32-zlib) [ 9307.378592] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 9307.655398] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 9309.287135] Key type lgssc registered [ 9310.476902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9320.402688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9320.418446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.936192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.141837] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9331.173821] LustreError: 321817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9331.201723] LustreError: 321817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9335.702332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.255038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9341.527145] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9352.441109] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9358.647671] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9366.458460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.765367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9373.561378] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9382.056935] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9391.586462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9391.591635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9391.599302] Lustre: Skipped 2 previous similar messages [ 9391.606443] Lustre: Skipped 2 previous similar messages [ 9395.385789] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.20@tcp (stopping) [ 9396.715527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9396.719559] Lustre: Skipped 2 previous similar messages [ 9396.870223] Lustre: server umount lustre-MDT0000 complete [ 9400.458322] LustreError: 324312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.976087] LustreError: 322552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622871 with bad export cookie 1185743625582886656 [ 9400.991035] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9401.001312] LustreError: 322552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9401.824865] LustreError: 323660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.843394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.868797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9405.586755] LustreError: 321812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9405.868534] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9405.875994] Lustre: Skipped 1 previous similar message [ 9407.656206] Lustre: server umount lustre-MDT0001 complete [ 9415.000842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9423.189134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.376570] LustreError: 325397:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9425.377988] LustreError: 321125:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004dda2b13 x1874458342106112/t0(0) o250->MGC192.168.206.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9425.731321] LustreError: 323650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9425.746305] LustreError: 323650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9425.805961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.847597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9430.800896] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9435.060762] LustreError: 325411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.069652] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9435.079042] LustreError: 325411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9441.219670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.421655] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.20@tcp (not set up) [ 9441.706371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9446.904931] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9447.383820] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9454.801660] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9456.885247] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9463.254817] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.879189] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9471.968774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9471.977913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9471.994547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9472.496278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9484.255236] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9484.433883] Lustre: server umount lustre-OST0000 complete [ 9492.965363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9492.966173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9492.973314] Lustre: Skipped 2 previous similar messages [ 9494.312265] Lustre: server umount lustre-MDT0000 complete [ 9498.081562] LustreError: 326900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9498.106550] LustreError: 326900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9498.587845] LustreError: 321799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622969 with bad export cookie 1185743625582894363 [ 9498.589451] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9498.600185] LustreError: 321799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9499.112429] Lustre: server umount lustre-MDT0001 complete [ 9510.734160] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9513.547302] Key type lgssc unregistered [ 9513.871586] LNet: 328091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9513.882990] LNetError: 328091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9513.900991] LNet: Removed LNI 192.168.206.120@tcp [ 9514.596853] Key type .llcrypt unregistered [ 9514.599354] Key type ._llcrypt unregistered [ 9538.456340] Key type ._llcrypt registered [ 9538.458342] Key type .llcrypt registered [ 9538.561834] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 9539.630303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9539.717648] alg: No test for adler32 (adler32-zlib) [ 9540.735590] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 9540.881213] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 9542.495155] Key type lgssc registered [ 9543.719215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9551.871795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9551.904769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9553.527739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9559.015258] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9563.112667] LustreError: 329506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.136638] LustreError: 329506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9568.233106] LustreError: 329525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9569.015229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.387298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9574.122584] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9583.451075] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9591.545750] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9599.456511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.459495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9599.473580] Lustre: Skipped 1 previous similar message [ 9599.492273] Lustre: Skipped 1 previous similar message [ 9601.575487] Lustre: server umount lustre-MDT0000 complete [ 9604.576874] LustreError: 329525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9604.585135] LustreError: 329525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9604.786478] LustreError: 329486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623075 with bad export cookie 14276669053573413528 [ 9604.796646] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9605.014329] Lustre: server umount lustre-MDT0001 complete [ 9619.607232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.830533] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9619.869840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9620.033508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9624.257120] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9632.025154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9632.107058] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9632.121801] Lustre: 332098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9632.332422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9636.377340] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9643.059106] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9648.683767] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9658.696947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9658.990191] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9659.007510] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9659.015732] LustreError: 333983:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9659.021988] LustreError: 333983:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9659.026961] LustreError: 333983:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9659.098954] Lustre: server umount lustre-OST0000 complete [ 9659.100479] LustreError: 333983:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9675.118845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9675.618670] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9675.990457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.236414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9685.713976] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9697.288972] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9708.473299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9708.694422] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9708.867851] Lustre: lustre-OST0001: new disk, initializing [ 9708.873180] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9708.886959] Lustre: 335528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9709.105365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9715.309362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9715.317720] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9715.663778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9717.189123] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9727.348791] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9731.047888] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9731.051504] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.052328] LustreError: Skipped 1 previous similar message [ 9731.060697] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9731.066845] Lustre: Skipped 1 previous similar message [ 9736.168290] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9736.176897] Lustre: Skipped 1 previous similar message [ 9736.795866] Lustre: server umount lustre-OST0001 complete [ 9741.282389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.288817] LustreError: 334452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9741.298466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9741.305310] Lustre: Skipped 2 previous similar messages [ 9741.331554] LustreError: 334452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9746.403850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9746.404307] LustreError: 334683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.412519] Lustre: Skipped 2 previous similar messages [ 9746.431746] LustreError: 334683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9747.457295] Lustre: server umount lustre-OST0000 complete [ 9751.529120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9751.548426] Lustre: Skipped 1 previous similar message [ 9756.640767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9756.652043] Lustre: Skipped 3 previous similar messages [ 9765.343108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9765.682626] Lustre: server umount lustre-MDT0000 complete [ 9766.883615] LustreError: 334652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.900167] LustreError: 334652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9770.350176] LustreError: 332090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623241 with bad export cookie 14276669053573414081 [ 9770.362370] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9770.365849] LustreError: 332090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9771.193517] Lustre: server umount lustre-MDT0001 complete [ 9782.997181] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9785.336770] Key type lgssc unregistered [ 9785.550428] LNet: 337491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9785.560028] LNetError: 337491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9785.571877] LNet: Removed LNI 192.168.206.120@tcp [ 9786.239140] Key type .llcrypt unregistered [ 9786.241339] Key type ._llcrypt unregistered [ 9859.905712] Key type ._llcrypt registered [ 9859.909971] Key type .llcrypt registered [ 9860.145256] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 9879.094251] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 9880.318246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9880.420719] alg: No test for adler32 (adler32-zlib) [ 9881.470826] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [ 9881.631814] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 9883.287159] Key type lgssc registered [ 9884.744534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9933.004793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9933.017787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9934.332554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9934.372653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9934.474955] Lustre: lustre-MDT0000: new disk, initializing [ 9934.589532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9934.616047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9939.937460] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9950.820124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.913223] Lustre: 342440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9950.949293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9950.952201] Lustre: Skipped 1 previous similar message [ 9951.008807] Lustre: lustre-MDT0001: new disk, initializing [ 9951.153202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9951.212224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9951.222582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9957.482726] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9967.887286] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9974.535974] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9980.695340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9980.895505] Lustre: lustre-OST0000: new disk, initializing [ 9980.897741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9980.900971] Lustre: 344408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9980.964446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.851879] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 9987.663452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9987.683227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9988.008200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9997.965494] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10002.445500] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10002.645065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10007.619491] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10008.017266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10013.671060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10013.673178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10013.740923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10017.826911] Lustre: server umount lustre-OST0000 complete [10023.908673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.927947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.940394] Lustre: Skipped 2 previous similar messages [10023.962045] Lustre: Skipped 2 previous similar messages [10029.029677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10029.041747] Lustre: Skipped 1 previous similar message [10029.800748] Lustre: server umount lustre-MDT0000 complete [10034.873961] LustreError: 342433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623505 with bad export cookie 11036470952066883504 [10034.880311] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10034.899907] LustreError: 342433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10035.911753] Lustre: server umount lustre-MDT0001 complete [10047.920656] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:05:17 (1787623517) [10055.925681] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:05:26 (1787623526) [10064.650064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10065.148746] LustreError: 347166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.167594] LustreError: 347166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10065.247408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10069.569352] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10070.500629] LustreError: 347167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.619063] LustreError: 347166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.015229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.250964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10082.268740] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10089.486942] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10095.503581] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10102.820866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10103.084064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10109.530831] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10118.169424] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10126.166902] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10128.039517] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10129.577462] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10131.009106] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10132.948068] Lustre: Failing over lustre-MDT0000 [10133.234641] Lustre: server umount lustre-MDT0000 complete [10134.498423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10134.503918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.519053] LustreError: 347166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.154078] LustreError: 347162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.170400] LustreError: 347162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10151.327151] Lustre: 340662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787623606/real 1787623606] req@0000000084c191e7 x1874458944019200/t0(0) o400->MGC192.168.206.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787623622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10151.359565] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10151.376853] LustreError: 347895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.403699] LustreError: 347895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10152.349319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.641456] LustreError: 340661:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f92e7232 x1874458944024448/t0(0) o250->MGC192.168.206.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10161.806676] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.20@tcp (not set up) [10162.064910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10162.129478] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10166.764354] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10166.932700] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10167.289629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10167.333520] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10167.385695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10177.494303] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10179.191433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10183.175547] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10185.422925] Lustre: Failing over lustre-MDT0000 [10185.883882] Lustre: server umount lustre-MDT0000 complete [10187.433611] LustreError: 347162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.449518] LustreError: 347162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10187.751247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10187.753369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.775267] Lustre: Skipped 3 previous similar messages [10204.130445] Lustre: 340662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787623658/real 1787623658] req@0000000041079f70 x1874458944040832/t0(0) o400->MGC192.168.206.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787623674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10204.202453] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10204.375240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.994047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10216.097475] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10217.638294] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10221.052836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10221.064114] Lustre: Skipped 2 previous similar messages [10221.123364] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10221.247284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10223.934885] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10235.137250] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10237.140937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10246.624150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10246.632352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10246.640522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10246.656318] Lustre: Skipped 3 previous similar messages [10250.620377] Lustre: server umount lustre-MDT0000 complete [10251.745460] LustreError: 347167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.778803] LustreError: 347167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages [10255.429741] LustreError: 347149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623726 with bad export cookie 11036470952066886220 [10255.431291] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10255.444807] LustreError: 347149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10255.804830] Lustre: server umount lustre-MDT0001 complete [10272.898469] Lustre: server umount lustre-OST0000 complete [10292.214495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.771129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10297.859879] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10308.166498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.599980] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10322.280987] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10329.031214] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10336.754390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10338.163134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10342.775174] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10352.537672] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10363.877068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10363.877323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.878181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.878186] Lustre: Skipped 2 previous similar messages [10363.884502] LustreError: Skipped 1 previous similar message [10363.914585] Lustre: Skipped 1 previous similar message [10368.839846] Lustre: server umount lustre-OST0000 complete [10379.427222] Lustre: server umount lustre-MDT0000 complete [10383.332803] LustreError: 354500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10383.353868] LustreError: 354500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10383.847261] LustreError: 353496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623854 with bad export cookie 11036470952066886955 [10383.848482] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10383.857960] LustreError: 353496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10384.325181] Lustre: server umount lustre-MDT0001 complete [10401.918293] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [10406.194687] Key type lgssc unregistered [10406.582065] LNet: 357052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10406.611967] LNetError: 357052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10406.636371] LNet: Removed LNI 192.168.206.120@tcp [10407.593885] Key type .llcrypt unregistered [10407.599709] Key type ._llcrypt unregistered [10441.079213] Key type ._llcrypt registered [10441.081465] Key type .llcrypt registered [10441.188953] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [10443.614791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10443.698432] alg: No test for adler32 (adler32-zlib) [10445.136749] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [10445.320939] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [10447.015336] Key type lgssc registered [10448.198777] Lustre: Echo OBD driver; http://www.lustre.org/ [10455.936455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10455.952470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10457.196993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10459.594978] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10464.273056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.555896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10467.090362] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10475.952398] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10485.235456] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10499.500691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10499.757915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10503.909450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10511.954531] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10526.189923] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10546.164207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.189646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10549.790198] Lustre: server umount lustre-MDT0000 complete [10551.269302] LustreError: 358572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.322367] LustreError: 358572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10556.403937] LustreError: 358572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.453646] LustreError: 358572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10558.545181] LustreError: 358553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624029 with bad export cookie 12780047711199578262 [10558.573212] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10558.580750] LustreError: 358553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10559.849326] Lustre: server umount lustre-MDT0001 complete [10577.837363] Lustre: server umount lustre-OST0000 complete [10595.402666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.837897] LustreError: 362306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10595.918307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10600.546423] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10600.931082] LustreError: 362307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.048405] LustreError: 362306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10608.873235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10609.258441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10613.241310] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10620.431229] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10626.237851] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10633.914102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10634.158387] LustreError: 364129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.431335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10641.424224] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10641.922339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10651.412707] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10665.442622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10665.446989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10665.461819] Lustre: Skipped 2 previous similar messages [10665.464538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10665.468880] Lustre: Skipped 2 previous similar messages [10667.490732] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10667.498731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10667.540575] Lustre: Skipped 2 previous similar messages [10670.497673] Lustre: server umount lustre-MDT0000 complete [10672.611637] LustreError: 363035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10672.637957] LustreError: 363035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10674.241823] LustreError: 362286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624145 with bad export cookie 12780047711199579354 [10674.246347] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10674.253632] LustreError: 362286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10674.558359] Lustre: server umount lustre-MDT0001 complete [10687.394229] Lustre: server umount lustre-OST0000 complete [10704.202714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.640657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10708.963857] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10709.984704] LustreError: 366221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.001750] LustreError: 366221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10718.637878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.767900] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10734.812102] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10741.772354] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10749.084717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10749.457746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10749.463713] Lustre: Skipped 1 previous similar message [10750.531086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10755.325503] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [10762.367755] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10768.867094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10768.874325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10768.897127] Lustre: Skipped 1 previous similar message [10768.901178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10773.867554] Lustre: server umount lustre-OST0000 complete [10779.104978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10779.108528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10779.125608] Lustre: Skipped 2 previous similar messages [10779.148774] Lustre: Skipped 2 previous similar messages [10784.225769] LustreError: 366220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.251224] LustreError: 366220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10784.280956] Lustre: server umount lustre-MDT0000 complete [10789.053878] LustreError: 366200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624260 with bad export cookie 12780047711199580383 [10789.070951] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10789.646311] Lustre: server umount lustre-MDT0001 complete [10801.712732] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [10804.872877] Key type lgssc unregistered [10805.204605] LNet: 369757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10805.218526] LNetError: 369757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10805.241782] LNet: Removed LNI 192.168.206.120@tcp [10806.236147] Key type .llcrypt unregistered [10806.241524] Key type ._llcrypt unregistered [10833.993972] Key type ._llcrypt registered [10833.996812] Key type .llcrypt registered [10834.121814] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [10835.317483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10835.330821] alg: No test for adler32 (adler32-zlib) [10836.483932] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [10836.893600] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [10838.551733] Key type lgssc registered [10839.796713] Lustre: Echo OBD driver; http://www.lustre.org/ [10848.752172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10848.771121] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10848.776833] LustreError: 371198:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10848.808989] LustreError: 371198:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10848.817659] LustreError: 371198:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10848.826461] LustreError: 371198:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10848.838188] LustreError: 371198:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10857.676475] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10857.687526] LustreError: 371595:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10857.711712] LustreError: 371595:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10857.717127] LustreError: 371595:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10857.722503] LustreError: 371595:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10857.728810] LustreError: 371595:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10880.863306] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [10884.172071] Key type lgssc unregistered [10884.450277] LNet: 372603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10884.466586] LNetError: 372603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10884.491489] LNet: Removed LNI 192.168.206.120@tcp [10885.523486] Key type .llcrypt unregistered [10885.526879] Key type ._llcrypt unregistered [10906.510874] Key type ._llcrypt registered [10906.512829] Key type .llcrypt registered [10906.599708] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [10925.650533] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [10927.493218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10927.514865] alg: No test for adler32 (adler32-zlib) [10929.039896] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [10929.432770] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [10931.303271] Key type lgssc registered [10933.311390] Lustre: Echo OBD driver; http://www.lustre.org/ [10980.890265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10980.940568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10982.205824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10982.229573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10982.317195] Lustre: lustre-MDT0000: new disk, initializing [10982.422359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10982.438366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10991.487524] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11012.172355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.534775] Lustre: 375760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11012.679035] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11012.692802] Lustre: Skipped 1 previous similar message [11012.890689] Lustre: lustre-MDT0001: new disk, initializing [11013.051462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11013.119774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11013.137368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11019.820176] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11030.958714] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11038.465484] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11049.209431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11049.868737] Lustre: lustre-OST0000: new disk, initializing [11049.873511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11049.892993] Lustre: 377734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11050.141740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11058.228183] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11059.275722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11059.280497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11059.362810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11070.789616] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11075.390475] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11075.548112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11079.954610] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11080.233135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11085.282818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11085.293113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11085.311138] LustreError: Skipped 1 previous similar message [11085.312549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11085.323152] Lustre: Skipped 1 previous similar message [11085.344947] Lustre: Skipped 1 previous similar message [11088.397790] Lustre: server umount lustre-OST0000 complete [11095.017777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11095.022565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.028983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11097.953819] Lustre: server umount lustre-MDT0000 complete [11101.983135] LustreError: 379278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624572 with bad export cookie 15223830045386464212 [11102.000144] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11102.000354] LustreError: 379278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11102.682771] Lustre: server umount lustre-MDT0001 complete [11113.911645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11115.863808] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:23:05 (1787624585) [11122.907399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.508805] LustreError: 380197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11123.531183] LustreError: 380197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11123.650202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11127.957501] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11128.803689] LustreError: 380198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.931820] LustreError: 380197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.912660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11137.403223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11141.713533] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11149.975317] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11156.060788] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11163.310693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.689837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11169.738963] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11177.569841] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11183.961525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11184.157483] Lustre: lustre-OST0001: new disk, initializing [11184.163612] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11184.175669] Lustre: 383034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11184.278876] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11190.107115] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11190.418537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11190.431234] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11190.566465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11199.744137] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11206.115637] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11206.124369] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.132707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11206.140165] Lustre: Skipped 2 previous similar messages [11206.141213] Lustre: Skipped 1 previous similar message [11209.684065] Lustre: server umount lustre-OST0001 complete [11211.235876] LustreError: 383416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11211.265880] LustreError: 383416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11216.359474] LustreError: 382017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11216.369552] LustreError: 382017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11226.598480] LustreError: 382019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11226.612103] LustreError: 382019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11229.152274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11229.165903] LustreError: Skipped 1 previous similar message [11229.172067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11229.188924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11229.203016] Lustre: Skipped 2 previous similar messages [11233.489379] Lustre: server umount lustre-OST0000 complete [11239.912598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11239.933387] LustreError: Skipped 1 previous similar message [11239.948283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.981936] Lustre: Skipped 1 previous similar message [11240.012560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11240.031854] Lustre: Skipped 3 previous similar messages [11246.123730] Lustre: server umount lustre-MDT0000 complete [11247.075797] LustreError: 380925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11247.105071] LustreError: 380925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11251.067989] LustreError: 383032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624722 with bad export cookie 15223830045386465094 [11251.068978] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11251.074477] LustreError: 383032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11257.637848] Lustre: server umount lustre-MDT0001 complete [11267.912090] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [11271.048875] Key type lgssc unregistered [11271.345477] LNet: 385027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11271.359690] LNetError: 385027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11271.383362] LNet: Removed LNI 192.168.206.120@tcp [11272.262174] Key type .llcrypt unregistered [11272.266941] Key type ._llcrypt unregistered [11320.229474] Key type ._llcrypt registered [11320.233025] Key type .llcrypt registered [11320.330129] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [11321.867822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11321.925545] alg: No test for adler32 (adler32-zlib) [11323.101390] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [11323.456518] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [11325.159197] Key type lgssc registered [11326.524243] Lustre: Echo OBD driver; http://www.lustre.org/ [11334.726528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11334.761423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.104591] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11336.156211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11336.404348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11341.574509] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11349.974688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.101356] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11350.133438] Lustre: 387819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11350.402015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11355.927033] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11363.144257] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11368.358434] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11370.976477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11370.982305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11370.999147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11373.027549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11373.035753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11376.292864] Lustre: server umount lustre-MDT0000 complete [11379.845300] LustreError: 387812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624850 with bad export cookie 4393076796480818830 [11379.849205] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11379.857853] LustreError: 387812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11380.155591] Lustre: server umount lustre-MDT0001 complete [11390.001447] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:27:40 (1787624860) [11397.193254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11397.741385] LustreError: 390335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.755993] LustreError: 390335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11397.795342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11402.434670] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11403.233885] LustreError: 390336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.357829] LustreError: 390335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11411.348520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.794854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11416.392988] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11424.701715] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11430.563223] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11437.922187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11438.173149] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11438.367758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11444.882619] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11452.704562] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11459.894712] Lustre: Permanently deactivating lustre-OST0000 [11459.908604] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11467.260200] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11469.006779] LustreError: 393322:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11469.016694] LustreError: 393322:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11469.022619] LustreError: 393322:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11471.751216] LustreError: 393371:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11471.757626] LustreError: 393371:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11479.543213] Lustre: server umount lustre-OST0000 complete [11482.215381] LustreError: 393620:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11482.219876] LustreError: 393620:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11485.666215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11485.676203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11485.688482] Lustre: Skipped 1 previous similar message [11485.718568] Lustre: Skipped 2 previous similar messages [11490.785202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11490.789082] Lustre: Skipped 1 previous similar message [11495.012801] Lustre: server umount lustre-MDT0000 complete [11495.909619] LustreError: 390332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.962679] LustreError: 390332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11499.055245] LustreError: 390315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624970 with bad export cookie 4393076796480819411 [11499.060621] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11499.064882] LustreError: 390315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11509.382592] Lustre: server umount lustre-MDT0001 complete [11518.156542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11521.636746] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11524.090505] LustreError: 394780:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11530.054440] Lustre: 395119:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.120@tcp [11531.978371] Lustre: server umount MGS complete [11545.565520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11546.052677] LustreError: 395843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11546.102141] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11546.116828] Lustre: Skipped 1 previous similar message [11546.206027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11552.463728] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11556.336462] LustreError: 395844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.385266] LustreError: 395844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11562.228152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11562.574906] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11562.714419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11567.448223] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11574.966879] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11580.233586] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11582.099171] Lustre: Permanently reactivating lustre-OST0000 [11582.111637] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11582.123276] Lustre: Skipped 2 previous similar messages [11611.105481] LustreError: 397630:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11618.425271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.969926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11625.497606] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11634.280661] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11640.800484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11640.812879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11640.828498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.833929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11646.418201] Lustre: server umount lustre-OST0000 complete [11652.066066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11652.070561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11652.078612] Lustre: Skipped 2 previous similar messages [11652.090379] Lustre: Skipped 3 previous similar messages [11664.351646] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11666.405692] LustreError: 395839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.435703] LustreError: 395839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11666.611969] Lustre: server umount lustre-MDT0000 complete [11670.354526] LustreError: 395823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625141 with bad export cookie 4393076796480820461 [11670.361664] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11670.379498] LustreError: 395823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11670.719563] Lustre: server umount lustre-MDT0001 complete [11682.017633] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [11684.969512] Key type lgssc unregistered [11685.303701] LNet: 400211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11685.320531] LNetError: 400211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11685.340749] LNet: Removed LNI 192.168.206.120@tcp [11686.123987] Key type .llcrypt unregistered [11686.126678] Key type ._llcrypt unregistered [11704.987361] Key type ._llcrypt registered [11704.989553] Key type .llcrypt registered [11705.124853] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [11718.957903] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [11719.984563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11720.024388] alg: No test for adler32 (adler32-zlib) [11721.064867] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [11721.302902] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [11722.952921] Key type lgssc registered [11723.870120] Lustre: Echo OBD driver; http://www.lustre.org/ [11753.659812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11753.672348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11754.883298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11754.911845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11754.995064] Lustre: lustre-MDT0000: new disk, initializing [11755.086091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11755.101163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11758.747410] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11768.454041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.599701] Lustre: 403368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11768.794145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11768.810969] Lustre: Skipped 1 previous similar message [11768.868117] Lustre: lustre-MDT0001: new disk, initializing [11769.054185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11769.119755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11769.135377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11774.252487] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11787.036839] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11794.796315] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11801.678551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11801.878313] Lustre: lustre-OST0000: new disk, initializing [11801.881422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11801.886268] Lustre: 405336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11801.966842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11807.231507] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11807.359161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11807.378129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11807.528488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11816.663707] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11821.286305] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11821.471426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11825.611648] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11825.782416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11828.194652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11828.201684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11828.207639] Lustre: Skipped 1 previous similar message [11828.208343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11828.233229] Lustre: Skipped 1 previous similar message [11833.314574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11833.326954] Lustre: Skipped 1 previous similar message [11833.842682] Lustre: server umount lustre-OST0000 complete [11838.433127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.469708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11843.556518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11843.564874] Lustre: Skipped 1 previous similar message [11843.735959] Lustre: server umount lustre-MDT0000 complete [11847.790854] LustreError: 403358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625318 with bad export cookie 17858890717984753973 [11847.791596] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11847.802737] LustreError: 403358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11848.250453] Lustre: server umount lustre-MDT0001 complete [11855.927411] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11857.493549] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:35:27 (1787625327) [11863.708471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11864.101076] LustreError: 407797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.114426] LustreError: 407797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11864.182486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11868.071451] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11869.671206] LustreError: 407798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.789071] LustreError: 407797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11876.351525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.825596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11881.947851] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11890.334825] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11895.660176] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11902.213939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11902.464047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11908.486415] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [11915.339195] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11916.860727] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11919.313048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11930.081607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11930.086681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.092722] Lustre: Skipped 1 previous similar message [11930.096062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11932.128449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11932.144938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11935.045799] Lustre: server umount lustre-OST0000 complete [11942.370855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11942.386695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11942.396816] Lustre: Skipped 1 previous similar message [11942.412398] Lustre: Skipped 3 previous similar messages [11944.982864] Lustre: server umount lustre-MDT0000 complete [11947.489083] LustreError: 407797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.543728] LustreError: 407797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11949.851071] LustreError: 407777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625420 with bad export cookie 17858890717984754862 [11949.852507] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11949.865085] LustreError: 407777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11950.146345] Lustre: server umount lustre-MDT0001 complete [11962.324101] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [11965.045941] Key type lgssc unregistered [11965.311044] LNet: 411525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11965.316522] LNetError: 411525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11965.330371] LNet: Removed LNI 192.168.206.120@tcp [11966.147098] Key type .llcrypt unregistered [11966.150090] Key type ._llcrypt unregistered [11989.661482] Key type ._llcrypt registered [11989.663749] Key type .llcrypt registered [11989.755173] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [11990.043630] Key type .llcrypt unregistered [11990.049658] Key type ._llcrypt unregistered [12007.316238] Key type ._llcrypt registered [12007.318750] Key type .llcrypt registered [12007.407797] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [12008.616684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12008.704256] alg: No test for adler32 (adler32-zlib) [12009.785442] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12009.982253] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [12011.631278] Key type lgssc registered [12012.875394] Lustre: Echo OBD driver; http://www.lustre.org/ [12019.253856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12019.268993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12020.650338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12024.395136] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12030.945973] LustreError: 413759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12030.969837] LustreError: 413759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12031.019210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12031.269657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12036.319632] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12036.582143] LustreError: 413758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.600564] LustreError: 413758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12041.699793] LustreError: 413755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12041.716350] LustreError: 413755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12042.936535] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12045.797325] LustreError: 413759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12045.815150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12045.820820] LustreError: 413759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12049.704357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12050.095205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12054.365356] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12065.761206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12065.770065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12065.774709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12065.796849] Lustre: Skipped 1 previous similar message [12067.921467] Lustre: server umount lustre-OST0000 complete [12075.492492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12075.497125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.502738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12075.508145] Lustre: Skipped 1 previous similar message [12077.781918] Lustre: server umount lustre-MDT0000 complete [12081.122673] LustreError: 413759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12081.148046] LustreError: 413759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12081.365626] LustreError: 413740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625552 with bad export cookie 322110174305297400 [12081.370222] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12081.381435] LustreError: 413740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12081.718720] Lustre: server umount lustre-MDT0001 complete [12092.838360] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [12095.431985] Key type lgssc unregistered [12095.686505] LNet: 416907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12095.695601] LNetError: 416907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12095.722780] LNet: Removed LNI 192.168.206.120@tcp [12096.559934] Key type .llcrypt unregistered [12096.562400] Key type ._llcrypt unregistered [12118.206908] Key type ._llcrypt registered [12118.211876] Key type .llcrypt registered [12118.313347] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [12119.128922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12119.168071] alg: No test for adler32 (adler32-zlib) [12120.182258] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12120.380637] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [12122.031203] Key type lgssc registered [12122.984484] Lustre: Echo OBD driver; http://www.lustre.org/ [12130.209937] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12130.226748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12145.759565] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12150.879336] LustreError: 418293:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.120@tcp: failed processing log, type 4: rc = -110 [12177.567401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12184.989504] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12196.414668] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12203.186968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.556330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12203.641533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12207.704417] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12208.611907] LustreError: 418318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.640520] LustreError: 418318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12212.709051] LustreError: 419329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12212.731732] LustreError: 419329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12215.643960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12215.887660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12219.539596] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12225.966654] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12230.776916] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12238.613919] Lustre: server umount lustre-OST0000 complete [12244.964992] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12244.976253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.995673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12246.501214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12246.502754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12246.518047] Lustre: Skipped 1 previous similar message [12251.621580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12251.628332] Lustre: Skipped 1 previous similar message [12251.693458] Lustre: server umount lustre-MDT0000 complete [12255.317528] LustreError: 418301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625726 with bad export cookie 16945654894054452008 [12255.327587] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12255.342564] LustreError: 418301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12266.225991] Lustre: server umount lustre-MDT0001 complete [12276.682934] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [12279.443893] Key type lgssc unregistered [12279.645829] LNet: 421868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12279.650449] LNetError: 421868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12280.679386] LNet: Removed LNI 192.168.206.120@tcp [12281.435900] Key type .llcrypt unregistered [12281.438505] Key type ._llcrypt unregistered [12303.307484] Key type ._llcrypt registered [12303.310249] Key type .llcrypt registered [12303.403756] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [12304.521254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12304.728303] alg: No test for adler32 (adler32-zlib) [12305.837713] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12306.165492] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [12307.920512] Key type lgssc registered [12308.971646] Lustre: Echo OBD driver; http://www.lustre.org/ [12316.120957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12316.135746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12317.390452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12321.467463] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12327.905773] LustreError: 423285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12327.918592] LustreError: 423285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12329.476859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12330.249355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12334.340993] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12342.319049] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12349.239581] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12356.043687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12356.412755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12362.565767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12363.144880] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12370.590630] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12375.987329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.994377] Lustre: Skipped 1 previous similar message [12375.998298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12376.016575] Lustre: Skipped 2 previous similar messages [12378.286101] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.20@tcp (stopping) [12381.154113] LustreError: 423282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.168928] LustreError: 423282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12381.208901] Lustre: server umount lustre-MDT0000 complete [12383.820136] LustreError: 425130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.20@tcp arrived at 1787625854 with bad export cookie 5056240037148142500 [12385.846965] LustreError: 423266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625856 with bad export cookie 5056240037148141611 [12385.852904] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12385.854050] LustreError: 423266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12386.273060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12386.274174] LustreError: 423282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.289029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12386.306753] LustreError: 423282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12390.304760] LustreError: 424026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12390.307931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12390.326233] LustreError: 424026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12392.125342] Lustre: server umount lustre-OST0000 complete [12395.446822] LustreError: 423282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12395.465328] LustreError: 423282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12398.525519] LustreError: 424014:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000093f879b1 x1874461486775680/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12398.790370] Lustre: server umount lustre-MDT0001 complete [12407.974903] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [12410.309943] Key type lgssc unregistered [12410.544486] LNet: 426973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12410.554039] LNetError: 426973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12410.582689] LNet: Removed LNI 192.168.206.120@tcp [12411.339243] Key type .llcrypt unregistered [12411.341233] Key type ._llcrypt unregistered [12435.125983] Key type ._llcrypt registered [12435.131074] Key type .llcrypt registered [12435.294818] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [12436.318327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12436.448226] alg: No test for adler32 (adler32-zlib) [12437.582898] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12437.803273] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [12439.447185] Key type lgssc registered [12440.893621] Lustre: Echo OBD driver; http://www.lustre.org/ [12449.038223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12449.094628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12450.934124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12455.440347] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12460.514699] LustreError: 428389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12460.521855] LustreError: 428389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12462.688251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12463.020481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12466.683578] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12472.840149] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12478.196888] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12485.018479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12485.378776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12490.839293] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12495.621249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12498.157516] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12504.032589] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12504.042456] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.069979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12505.251691] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.20@tcp (stopping) [12506.097684] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12508.147992] Lustre: server umount lustre-MDT0001 complete [12510.350075] LustreError: 429452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.321338] LustreError: 428389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.325704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12516.326645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12516.327983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12516.327988] Lustre: Skipped 2 previous similar messages [12516.345886] LustreError: 428389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12521.441026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12521.443692] LustreError: 430691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12522.331833] Lustre: server umount lustre-OST0000 complete [12526.571467] LustreError: 428389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12527.039597] Lustre: server umount lustre-MDT0000 complete [12539.457663] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [12541.998960] Key type lgssc unregistered [12542.248474] LNet: 432074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12542.256218] LNetError: 432074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12542.269594] LNet: Removed LNI 192.168.206.120@tcp [12543.104872] Key type .llcrypt unregistered [12543.106821] Key type ._llcrypt unregistered [12566.151124] Key type ._llcrypt registered [12566.153993] Key type .llcrypt registered [12566.252723] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [12579.255379] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [12579.875663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12579.979036] alg: No test for adler32 (adler32-zlib) [12580.902862] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12581.095314] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [12582.775196] Key type lgssc registered [12583.551266] Lustre: Echo OBD driver; http://www.lustre.org/ [12620.626150] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [12636.160072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12636.199352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12637.330214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12637.349516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12637.392727] Lustre: lustre-MDT0000: new disk, initializing [12637.462140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12637.493461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12643.105190] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12658.523476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12658.767547] Lustre: lustre-OST0000: new disk, initializing [12658.769788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12658.773178] Lustre: Skipped 1 previous similar message [12658.776591] Lustre: 437119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12658.825577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12664.123644] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12664.913796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12664.922569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12664.948860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12672.751897] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12686.724184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12686.828187] Lustre: 436101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12686.888904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12686.948511] Lustre: lustre-MDT0001: new disk, initializing [12687.045048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12687.076386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12687.092736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12691.029904] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12701.447930] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12701.779496] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12706.142685] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12706.392513] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12712.425096] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12712.440729] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.445732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12712.453661] Lustre: Skipped 1 previous similar message [12712.464539] Lustre: Skipped 1 previous similar message [12715.692308] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.20@tcp (stopping) [12717.538245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12717.545780] Lustre: Skipped 1 previous similar message [12717.950298] Lustre: server umount lustre-MDT0001 complete [12727.787741] LustreError: 436107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12727.810525] LustreError: 436107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12732.902087] LustreError: 436109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12732.931256] LustreError: 436109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12736.480447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12736.486456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12736.495844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12738.020019] LustreError: 436112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.139129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12743.143680] LustreError: 436113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.145094] Lustre: Skipped 1 previous similar message [12748.262232] LustreError: 436112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12749.279119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12749.419699] Lustre: server umount lustre-OST0000 complete [12767.200044] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12767.480721] Lustre: server umount lustre-MDT0000 complete [12780.488842] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [12783.428841] Key type lgssc unregistered [12783.751043] LNet: 440872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12783.770215] LNetError: 440872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12783.792583] LNet: Removed LNI 192.168.206.120@tcp [12784.744209] Key type .llcrypt unregistered [12784.747935] Key type ._llcrypt unregistered [12824.348921] Key type ._llcrypt registered [12824.351605] Key type .llcrypt registered [12824.483092] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [12824.841429] Key type .llcrypt unregistered [12824.844702] Key type ._llcrypt unregistered [12841.052358] Key type ._llcrypt registered [12841.054572] Key type .llcrypt registered [12841.162352] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [12841.945672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12842.053884] alg: No test for adler32 (adler32-zlib) [12843.059645] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [12843.214828] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [12844.847156] Key type lgssc registered [12845.695272] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.178283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12871.240063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12872.423077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12872.446468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12872.494487] Lustre: lustre-MDT0000: new disk, initializing [12872.615334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12872.632167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12876.039364] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12884.552238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12884.632525] Lustre: 444251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12884.706794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12884.713697] Lustre: Skipped 1 previous similar message [12884.792368] Lustre: lustre-MDT0001: new disk, initializing [12884.889265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12884.940223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12884.956792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12888.811929] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12896.807317] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12902.170396] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12908.886829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12909.073713] Lustre: lustre-OST0000: new disk, initializing [12909.075970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12909.079956] Lustre: 446219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12909.173603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12915.320598] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12916.759229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12916.765655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12916.867158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12925.013798] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12932.067119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12932.072731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12932.079511] Lustre: Skipped 2 previous similar messages [12932.089949] Lustre: Skipped 2 previous similar messages [12937.184822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12937.190935] Lustre: Skipped 2 previous similar messages [12937.661931] Lustre: server umount lustre-MDT0000 complete [12941.560457] LustreError: 444245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626412 with bad export cookie 2892340972657000080 [12941.564647] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12941.585022] LustreError: 444245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12942.090969] Lustre: server umount lustre-MDT0001 complete [12956.245213] Lustre: server umount lustre-OST0000 complete [12966.354850] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:53:56 (1787626436) [12973.280950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12973.739994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12977.209138] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12979.168871] LustreError: 448250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12979.183781] LustreError: 448250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12983.264235] LustreError: 448251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12984.279414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12984.515726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12988.589961] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [12995.964839] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13001.676548] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13022.707879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13022.888302] Lustre: Found index 0 for lustre-OST0000, updating log [13023.068073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13028.142141] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [13034.789205] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13044.591753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13050.857858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13050.871244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13050.884753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13058.017772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13058.022345] Lustre: Skipped 3 previous similar messages [13064.164349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13064.170971] Lustre: Skipped 1 previous similar message [13065.183756] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13065.300389] Lustre: server umount lustre-OST0000 complete [13080.547927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13080.552104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13080.565019] Lustre: Skipped 2 previous similar messages [13080.586997] Lustre: Skipped 1 previous similar message [13081.823407] LNet: 1 local NIs in recovery (showing 1): 192.168.206.120@tcp [13094.367159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13094.620857] Lustre: server umount lustre-MDT0000 complete [13095.905259] LustreError: 450068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13095.923512] LustreError: 450068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13097.971496] LustreError: 448232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787626568 with bad export cookie 2892340972657001473 [13097.975575] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13097.977486] LustreError: 448232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13098.244820] Lustre: server umount lustre-MDT0001 complete [13106.646518] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:56:16 (1787626576) [13158.623348] LNet: 1 local NIs in recovery (showing 1): 192.168.206.120@tcp [13717.193244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13717.540561] LustreError: 454698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13717.554459] LustreError: 454698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13717.598352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13721.016970] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [13722.592724] LustreError: 454699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13727.714348] LustreError: 454698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13728.024617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13728.309244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13731.507466] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [13737.174530] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13741.856875] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13760.755963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13761.168236] Lustre: Found index 0 for lustre-OST0000, updating log [13761.579472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13766.719598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13769.013714] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [13773.024240] LNet: 1 local NIs in recovery (showing 1): 192.168.206.120@tcp [13776.973865] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13779.755903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13779.774295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13779.781455] Lustre: Skipped 4 previous similar messages [13793.247158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13793.367428] Lustre: server umount lustre-OST0000 complete [13798.115176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13798.124534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13798.127296] Lustre: Skipped 1 previous similar message [13798.147248] Lustre: Skipped 1 previous similar message [13802.272856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13802.285351] Lustre: Skipped 7 previous similar messages [13803.317176] LustreError: 455426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13803.333574] LustreError: 455426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13803.371358] Lustre: server umount lustre-MDT0000 complete [13806.734440] LustreError: 456545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627277 with bad export cookie 2892340972657002558 [13806.743791] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13807.000589] Lustre: server umount lustre-MDT0001 complete [13835.491024] LNet: 1 local NIs in recovery (showing 1): 192.168.206.120@tcp [13837.621184] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [13852.114063] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [13895.284949] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:09:24 (1787627364) [13914.648346] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [13917.797124] Key type lgssc unregistered [13918.254560] LNet: 461333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13918.292988] LNetError: 461333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14021.736039] LNet: Removed LNI 192.168.206.120@tcp [14022.716888] Key type .llcrypt unregistered [14022.721246] Key type ._llcrypt unregistered [14041.438569] Key type ._llcrypt registered [14041.442514] Key type .llcrypt registered [14041.561156] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [14042.730333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14042.760333] alg: No test for adler32 (adler32-zlib) [14043.834275] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [14044.126678] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [14045.847326] Key type lgssc registered [14046.876759] Lustre: Echo OBD driver; http://www.lustre.org/ [14050.972995] LNet: Added LNI 192.168.206.120@tcp100 [8/256/0/180] [14051.696052] LNet: Added LNI 192.168.206.120@tcp101 [8/256/0/180] [14052.460570] LNet: Added LNI 192.168.206.120@tcp102 [8/256/0/180] [14053.270466] LNet: Added LNI 192.168.206.120@tcp103 [8/256/0/180] [14054.022560] LNet: Added LNI 192.168.206.120@tcp104 [8/256/0/180] [14054.826754] LNet: Added LNI 192.168.206.120@tcp105 [8/256/0/180] [14055.604471] LNet: Added LNI 192.168.206.120@tcp106 [8/256/0/180] [14056.419377] LNet: Added LNI 192.168.206.120@tcp107 [8/256/0/180] [14057.204931] LNet: Added LNI 192.168.206.120@tcp108 [8/256/0/180] [14058.041153] LNet: Added LNI 192.168.206.120@tcp109 [8/256/0/180] [14058.762954] LNet: Added LNI 192.168.206.120@tcp110 [8/256/0/180] [14059.582994] LNet: Added LNI 192.168.206.120@tcp111 [8/256/0/180] [14060.513417] LNet: Added LNI 192.168.206.120@tcp112 [8/256/0/180] [14061.419801] LNet: Added LNI 192.168.206.120@tcp113 [8/256/0/180] [14062.223679] LNet: Added LNI 192.168.206.120@tcp114 [8/256/0/180] [14063.049773] LNet: Added LNI 192.168.206.120@tcp115 [8/256/0/180] [14063.960536] LNet: Added LNI 192.168.206.120@tcp116 [8/256/0/180] [14064.872707] LNet: Added LNI 192.168.206.120@tcp117 [8/256/0/180] [14065.576440] LNet: Added LNI 192.168.206.120@tcp118 [8/256/0/180] [14066.397763] LNet: Added LNI 192.168.206.120@tcp119 [8/256/0/180] [14067.262391] LNet: Added LNI 192.168.206.120@tcp120 [8/256/0/180] [14068.089942] LNet: Added LNI 192.168.206.120@tcp121 [8/256/0/180] [14068.921169] LNet: Added LNI 192.168.206.120@tcp122 [8/256/0/180] [14069.762293] LNet: Added LNI 192.168.206.120@tcp123 [8/256/0/180] [14070.497028] LNet: Added LNI 192.168.206.120@tcp124 [8/256/0/180] [14071.308733] LNet: Added LNI 192.168.206.120@tcp125 [8/256/0/180] [14072.275438] LNet: Added LNI 192.168.206.120@tcp126 [8/256/0/180] [14073.150728] LNet: Added LNI 192.168.206.120@tcp127 [8/256/0/180] [14074.177879] LNet: Added LNI 192.168.206.120@tcp128 [8/256/0/180] [14075.026918] LNet: Added LNI 192.168.206.120@tcp129 [8/256/0/180] [14075.857818] LNet: Added LNI 192.168.206.120@tcp130 [8/256/0/180] [14076.840707] LNet: Added LNI 192.168.206.120@tcp131 [8/256/0/180] [14077.729573] LNet: Added LNI 192.168.206.120@tcp132 [8/256/0/180] [14078.549447] LNet: Added LNI 192.168.206.120@tcp133 [8/256/0/180] [14079.284602] LNet: Added LNI 192.168.206.120@tcp134 [8/256/0/180] [14080.017872] LNet: Added LNI 192.168.206.120@tcp135 [8/256/0/180] [14086.708492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14086.719892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14087.850545] Lustre: 464364:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.120@tcp: can't add NID 192.168.206.120@tcp130: rc = -75 [14087.952996] LNetError: 464387:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14087.976605] LNetError: Protocol error connecting to 192.168.206.120@tcp120 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14087.997060] LNetError: Skipped 7 previous similar messages [14088.082988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14088.124774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14088.172536] LNet: 8 local NIs in recovery (showing 8): 192.168.206.120@tcp121, 192.168.206.120@tcp125, 192.168.206.120@tcp126, 192.168.206.120@tcp127, 192.168.206.120@tcp124, 192.168.206.120@tcp128, 192.168.206.120@tcp129, 192.168.206.120@tcp120 [14088.210854] Lustre: lustre-MDT0000: new disk, initializing [14088.263536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14088.279201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14089.251576] LNetError: 464393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14089.264279] LNetError: 464393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [14089.276455] LNetError: Protocol error connecting to 192.168.206.120@tcp120 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14089.287209] LNetError: Skipped 12 previous similar messages [14090.273496] LNetError: 461822:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14090.288878] LNetError: Protocol error connecting to 192.168.206.120@tcp121 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14090.297744] LNetError: Skipped 2 previous similar messages [14092.240952] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14094.371203] LNetError: 464393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14094.380737] LNetError: 464393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14094.385470] LNetError: Protocol error connecting to 192.168.206.120@tcp121 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14094.396674] LNetError: Skipped 4 previous similar messages [14101.790717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14102.003157] Lustre: 464381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14102.084382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14102.094412] Lustre: Skipped 1 previous similar message [14102.246196] Lustre: lustre-MDT0001: new disk, initializing [14102.279893] Lustre: 465181:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.120@tcp: can't add NID 192.168.206.120@tcp130: rc = -75 [14102.290943] Lustre: 465181:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14102.397372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14102.429552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14102.439504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14102.560906] LNetError: 461824:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14102.565534] LNetError: 461824:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14102.573745] LNetError: Protocol error connecting to 192.168.206.120@tcp121 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14102.587324] LNetError: Skipped 1 previous similar message [14105.534367] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14115.013831] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14118.945533] LNetError: 461822:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14118.954335] LNetError: 461822:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14118.977882] LNetError: Protocol error connecting to 192.168.206.120@tcp121 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14118.996413] LNetError: Skipped 1 previous similar message [14121.008923] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14129.431381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14129.622304] Lustre: lustre-OST0000: new disk, initializing [14129.632284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14129.640559] Lustre: 466351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14129.727441] Lustre: 466357:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.120@tcp: can't add NID 192.168.206.120@tcp130: rc = -75 [14129.735506] Lustre: 466357:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14129.796700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14136.298552] LNetError: 461822:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14136.326918] LNetError: 461822:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14136.340480] LNetError: Protocol error connecting to 192.168.206.120@tcp105 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14136.363818] LNetError: Skipped 10 previous similar messages [14137.394119] Lustre: 466682:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.120@tcp: can't add NID 192.168.206.120@tcp130: rc = -75 [14137.411356] Lustre: 466682:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14137.484429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14137.515170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14137.777836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14137.872502] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14148.640484] LNet: 2 local NIs in recovery (showing 2): 192.168.206.120@tcp120, 192.168.206.120@tcp121 [14150.092289] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14158.308386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14158.309815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14158.311763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14158.327091] Lustre: Skipped 1 previous similar message [14163.406949] Lustre: server umount lustre-OST0000 complete [14168.548695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14168.550411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14168.561489] Lustre: Skipped 1 previous similar message [14168.584616] Lustre: Skipped 2 previous similar messages [14169.120706] LNetError: 461823:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.120 [14169.128787] LNetError: 461823:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [14169.141759] LNetError: Protocol error connecting to 192.168.206.120@tcp125 at host 192.168.206.120:988: is it running a compatible version of Lustre? [14169.153787] LNetError: Skipped 15 previous similar messages [14173.582702] Lustre: server umount lustre-MDT0000 complete [14177.001940] LustreError: 464373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787627647 with bad export cookie 3830933885701073919 [14177.003696] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14177.009936] LustreError: 464373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14177.308948] Lustre: server umount lustre-MDT0001 complete [14188.612719] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [14191.668654] Key type lgssc unregistered [14192.075901] LNet: 468268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14192.093705] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14192.101047] LNet: Removed LNI 192.168.206.120@tcp [14193.183759] LNet: Removed LNI 192.168.206.120@tcp100 [14193.191880] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14193.207541] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14193.219443] LNet: Removed LNI 192.168.206.120@tcp101 [14194.271774] LNet: Removed LNI 192.168.206.120@tcp102 [14194.275472] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14194.286729] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14194.294424] LNet: Removed LNI 192.168.206.120@tcp103 [14195.359156] LNet: Removed LNI 192.168.206.120@tcp104 [14196.383282] LNet: Removed LNI 192.168.206.120@tcp105 [14196.387889] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14196.400310] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14196.417222] LNet: Removed LNI 192.168.206.120@tcp106 [14197.474576] LNet: Removed LNI 192.168.206.120@tcp107 [14198.495610] LNet: Removed LNI 192.168.206.120@tcp108 [14199.519364] LNet: Removed LNI 192.168.206.120@tcp109 [14200.543134] LNet: Removed LNI 192.168.206.120@tcp110 [14200.551076] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14200.573075] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14200.581032] LNet: Removed LNI 192.168.206.120@tcp111 [14201.631346] LNet: Removed LNI 192.168.206.120@tcp112 [14202.655687] LNet: Removed LNI 192.168.206.120@tcp113 [14203.679209] LNet: Removed LNI 192.168.206.120@tcp114 [14204.703134] LNet: Removed LNI 192.168.206.120@tcp115 [14205.730387] LNet: Removed LNI 192.168.206.120@tcp116 [14206.752100] LNet: Removed LNI 192.168.206.120@tcp117 [14207.775175] LNet: Removed LNI 192.168.206.120@tcp118 [14208.799339] LNet: Removed LNI 192.168.206.120@tcp119 [14208.802700] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14208.812193] LNetError: 468268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14208.823420] LNet: Removed LNI 192.168.206.120@tcp120 [14208.828402] LNet: Removed LNI 192.168.206.120@tcp121 [14209.887142] LNet: Removed LNI 192.168.206.120@tcp122 [14210.911830] LNet: Removed LNI 192.168.206.120@tcp123 [14211.935868] LNet: Removed LNI 192.168.206.120@tcp124 [14212.959266] LNet: Removed LNI 192.168.206.120@tcp125 [14213.983549] LNet: Removed LNI 192.168.206.120@tcp126 [14215.007098] LNet: Removed LNI 192.168.206.120@tcp127 [14216.031135] LNet: Removed LNI 192.168.206.120@tcp128 [14217.055160] LNet: Removed LNI 192.168.206.120@tcp129 [14217.057221] LNet: Removed LNI 192.168.206.120@tcp130 [14217.065055] LNet: Removed LNI 192.168.206.120@tcp131 [14217.071136] LNet: Removed LNI 192.168.206.120@tcp132 [14217.073950] LNet: Removed LNI 192.168.206.120@tcp133 [14217.080286] LNet: Removed LNI 192.168.206.120@tcp134 [14217.110671] LNet: Removed LNI 192.168.206.120@tcp135 [14217.928611] Key type .llcrypt unregistered [14217.931211] Key type ._llcrypt unregistered [14240.183406] Key type ._llcrypt registered [14240.185930] Key type .llcrypt registered [14240.329670] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [14255.431749] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [14256.318896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14256.392925] alg: No test for adler32 (adler32-zlib) [14257.473291] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [14257.656541] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [14259.351271] Key type lgssc registered [14260.433324] Lustre: Echo OBD driver; http://www.lustre.org/ [14299.130229] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:16:08 (1787627768) [14317.044993] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [14320.154570] Key type lgssc unregistered [14320.581567] LNet: 472361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14320.590735] LNetError: 472361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14320.611725] LNet: Removed LNI 192.168.206.120@tcp [14321.563515] Key type .llcrypt unregistered [14321.566292] Key type ._llcrypt unregistered [14340.364485] Key type ._llcrypt registered [14340.371565] Key type .llcrypt registered [14340.491218] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [14341.681411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14341.708233] alg: No test for adler32 (adler32-zlib) [14342.766692] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [14342.960683] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [14344.615169] Key type lgssc registered [14345.641453] Lustre: Echo OBD driver; http://www.lustre.org/ [14354.607940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14354.630832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14355.914794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14355.962167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14356.038416] Lustre: lustre-MDT0000: new disk, initializing [14356.112303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14356.132479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14360.483678] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14370.636511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14370.767132] Lustre: 473575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14370.839040] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14370.845283] Lustre: Skipped 1 previous similar message [14370.933230] Lustre: lustre-MDT0001: new disk, initializing [14371.036791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14371.074403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14371.087321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14375.295859] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14383.819328] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14389.542710] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14397.338082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14397.874510] Lustre: lustre-OST0000: new disk, initializing [14397.878126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14397.883694] Lustre: 475542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14398.015837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14403.169750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14403.180094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14403.313953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14405.723540] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14416.057377] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14420.400645] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing lnet_if_list [14426.068854] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14426.100351] LNet: Added LNI 192.168.206.120@tcp10 [8/256/0/180] [14431.706440] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14431.746501] LNet: Added LNI 192.168.206.120@tcp11 [8/256/0/180] [14436.232193] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14436.280576] LNet: Added LNI 192.168.206.120@tcp12 [8/256/0/180] [14441.277943] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14441.302872] LNet: Added LNI 192.168.206.120@tcp13 [8/256/0/180] [14446.163668] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14446.197169] LNet: Added LNI 192.168.206.120@tcp14 [8/256/0/180] [14451.384901] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14451.441462] LNet: Added LNI 192.168.206.120@tcp15 [8/256/0/180] [14456.597440] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14456.622507] LNet: Added LNI 192.168.206.120@tcp16 [8/256/0/180] [14461.626404] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14461.662502] LNet: Added LNI 192.168.206.120@tcp17 [8/256/0/180] [14466.846285] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14466.888427] LNet: Added LNI 192.168.206.120@tcp18 [8/256/0/180] [14472.083494] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14472.114549] LNet: Added LNI 192.168.206.120@tcp19 [8/256/0/180] [14477.713878] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14477.737735] LNet: Added LNI 192.168.206.120@tcp20 [8/256/0/180] [14483.525517] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14483.552381] LNet: Added LNI 192.168.206.120@tcp21 [8/256/0/180] [14489.386647] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14489.428226] LNet: Added LNI 192.168.206.120@tcp22 [8/256/0/180] [14494.594513] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14494.633892] LNet: Added LNI 192.168.206.120@tcp23 [8/256/0/180] [14500.821728] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14500.885579] LNet: Added LNI 192.168.206.120@tcp24 [8/256/0/180] [14507.026560] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14507.072693] LNet: Added LNI 192.168.206.120@tcp25 [8/256/0/180] [14512.143131] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14512.183354] LNet: Added LNI 192.168.206.120@tcp26 [8/256/0/180] [14519.384176] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14519.417139] LNet: Added LNI 192.168.206.120@tcp27 [8/256/0/180] [14525.831136] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14525.900059] LNet: Added LNI 192.168.206.120@tcp28 [8/256/0/180] [14532.403242] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14532.451218] LNet: Added LNI 192.168.206.120@tcp29 [8/256/0/180] [14538.745830] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14538.767945] LNet: Added LNI 192.168.206.120@tcp30 [8/256/0/180] [14545.420905] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14545.448566] LNet: Added LNI 192.168.206.120@tcp31 [8/256/0/180] [14553.457578] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14553.529316] LNet: Added LNI 192.168.206.120@tcp32 [8/256/0/180] [14563.207291] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14563.270534] LNet: Added LNI 192.168.206.120@tcp33 [8/256/0/180] [14570.642721] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14570.692834] LNet: Added LNI 192.168.206.120@tcp34 [8/256/0/180] [14577.975462] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14577.998206] LNet: Added LNI 192.168.206.120@tcp35 [8/256/0/180] [14584.590962] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14584.632211] LNet: Added LNI 192.168.206.120@tcp36 [8/256/0/180] [14591.341216] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14591.373184] LNet: Added LNI 192.168.206.120@tcp37 [8/256/0/180] [14598.277164] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14598.307071] LNet: Added LNI 192.168.206.120@tcp38 [8/256/0/180] [14606.062492] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14606.120291] LNet: Added LNI 192.168.206.120@tcp39 [8/256/0/180] [14614.049156] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14614.075958] LNet: Added LNI 192.168.206.120@tcp40 [8/256/0/180] [14622.925891] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14622.942635] LNet: Added LNI 192.168.206.120@tcp41 [8/256/0/180] [14630.417651] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14630.449088] LNet: Added LNI 192.168.206.120@tcp42 [8/256/0/180] [14640.222368] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14640.266886] LNet: Added LNI 192.168.206.120@tcp43 [8/256/0/180] [14648.449966] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14648.512726] LNet: Added LNI 192.168.206.120@tcp44 [8/256/0/180] [14657.418566] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14657.456527] LNet: Added LNI 192.168.206.120@tcp45 [8/256/0/180] [14665.750610] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14665.814713] LNet: Added LNI 192.168.206.120@tcp46 [8/256/0/180] [14673.996587] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14674.025389] LNet: Added LNI 192.168.206.120@tcp47 [8/256/0/180] [14682.457392] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14682.491899] LNet: Added LNI 192.168.206.120@tcp48 [8/256/0/180] [14692.006722] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14692.068794] LNet: Added LNI 192.168.206.120@tcp49 [8/256/0/180] [14700.858337] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14700.877331] LNet: Added LNI 192.168.206.120@tcp50 [8/256/0/180] [14720.999473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14721.003432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14721.022698] Lustre: Skipped 1 previous similar message [14721.029219] Lustre: Skipped 2 previous similar messages [14726.112524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14726.134145] Lustre: Skipped 2 previous similar messages [14726.773590] Lustre: server umount lustre-MDT0000 complete [14730.763252] LustreError: 473566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628201 with bad export cookie 16547151039099076023 [14730.770216] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14730.773175] LustreError: 473566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14731.169674] Lustre: server umount lustre-MDT0001 complete [14747.638346] Lustre: 472847:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628202/real 1787628202] req@0000000019413559 x1874463621819264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787628218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14747.701080] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14747.712941] Lustre: Skipped 1 previous similar message [14750.854257] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [14751.720701] Lustre: 472845:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628206/real 1787628206] req@0000000038c21c8a x1874463621819392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787628222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14752.735591] Lustre: 472844:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628207/real 1787628207] req@00000000b5b45c4c x1874463621819520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787628223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14756.897536] Lustre: 472847:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628211/real 1787628211] req@00000000b0b5dd60 x1874463621819776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787628227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14761.056907] Lustre: 472847:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787628216/real 1787628216] req@00000000d4105fbc x1874463621819904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787628232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14761.462262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14775.392879] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a34a33123c880d [14775.398940] Lustre: MGC192.168.206.120@tcp: Connection restored to 0@lo (at 0@lo) [14775.872617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14776.887134] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14780.724782] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14782.952855] LustreError: 475536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14782.974424] LustreError: 475536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14788.066135] LustreError: 475537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14791.780190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14792.272709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14796.967189] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14797.288940] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14804.820544] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14812.295422] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14836.226745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14836.240073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14836.246042] Lustre: Skipped 1 previous similar message [14836.264729] Lustre: Skipped 1 previous similar message [14840.789846] Lustre: server umount lustre-OST0000 complete [14846.438160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14846.458548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14846.460524] Lustre: Skipped 1 previous similar message [14846.479565] Lustre: Skipped 1 previous similar message [14850.646839] Lustre: server umount lustre-MDT0000 complete [14851.552496] LustreError: 499786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14851.569013] LustreError: 499786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14854.625858] LustreError: 473566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628325 with bad export cookie 16547151039099078669 [14854.627429] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14854.643886] LustreError: 473566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14854.966898] Lustre: server umount lustre-MDT0001 complete [14865.502554] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [14867.983125] Key type lgssc unregistered [14868.282679] LNet: 502589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14868.288857] LNetError: 502589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14868.296933] LNet: Removed LNI 192.168.206.120@tcp [14868.299262] LNet: Removed LNI 192.168.206.120@tcp10 [14868.302407] LNet: Removed LNI 192.168.206.120@tcp11 [14868.304716] LNet: Removed LNI 192.168.206.120@tcp12 [14868.307467] LNet: Removed LNI 192.168.206.120@tcp13 [14868.309447] LNet: Removed LNI 192.168.206.120@tcp14 [14868.311873] LNet: Removed LNI 192.168.206.120@tcp15 [14868.315746] LNet: Removed LNI 192.168.206.120@tcp16 [14868.321573] LNet: Removed LNI 192.168.206.120@tcp17 [14868.325517] LNet: Removed LNI 192.168.206.120@tcp18 [14868.329715] LNet: Removed LNI 192.168.206.120@tcp19 [14868.331758] LNet: Removed LNI 192.168.206.120@tcp20 [14868.334542] LNet: Removed LNI 192.168.206.120@tcp21 [14868.337347] LNet: Removed LNI 192.168.206.120@tcp22 [14868.340288] LNet: Removed LNI 192.168.206.120@tcp23 [14868.343331] LNet: Removed LNI 192.168.206.120@tcp24 [14868.344922] LNet: Removed LNI 192.168.206.120@tcp25 [14868.347683] LNet: Removed LNI 192.168.206.120@tcp26 [14868.350748] LNet: Removed LNI 192.168.206.120@tcp27 [14868.352911] LNet: Removed LNI 192.168.206.120@tcp28 [14868.356706] LNet: Removed LNI 192.168.206.120@tcp29 [14868.362384] LNet: Removed LNI 192.168.206.120@tcp30 [14868.370194] LNet: Removed LNI 192.168.206.120@tcp31 [14868.375052] LNet: Removed LNI 192.168.206.120@tcp32 [14868.376737] LNet: Removed LNI 192.168.206.120@tcp33 [14868.382814] LNet: Removed LNI 192.168.206.120@tcp34 [14868.385549] LNet: Removed LNI 192.168.206.120@tcp35 [14868.387921] LNet: Removed LNI 192.168.206.120@tcp36 [14868.390582] LNet: Removed LNI 192.168.206.120@tcp37 [14868.396376] LNet: Removed LNI 192.168.206.120@tcp38 [14868.401615] LNet: Removed LNI 192.168.206.120@tcp39 [14868.406347] LNet: Removed LNI 192.168.206.120@tcp40 [14868.411680] LNet: Removed LNI 192.168.206.120@tcp41 [14868.413232] LNet: Removed LNI 192.168.206.120@tcp42 [14868.423566] LNet: Removed LNI 192.168.206.120@tcp43 [14868.428485] LNet: Removed LNI 192.168.206.120@tcp44 [14868.431975] LNet: Removed LNI 192.168.206.120@tcp45 [14868.437500] LNet: Removed LNI 192.168.206.120@tcp46 [14868.440516] LNet: Removed LNI 192.168.206.120@tcp47 [14868.443333] LNet: Removed LNI 192.168.206.120@tcp48 [14868.446495] LNet: Removed LNI 192.168.206.120@tcp49 [14868.459706] LNet: Removed LNI 192.168.206.120@tcp50 [14869.491776] Key type .llcrypt unregistered [14869.503245] Key type ._llcrypt unregistered [14893.287899] Key type ._llcrypt registered [14893.289812] Key type .llcrypt registered [14893.428402] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [14908.349687] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [14909.611330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14909.761587] alg: No test for adler32 (adler32-zlib) [14910.901795] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [14911.202374] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [14912.911205] Key type lgssc registered [14914.170733] Lustre: Echo OBD driver; http://www.lustre.org/ [14950.843575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14950.868213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14952.022810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14952.061397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14952.161640] Lustre: lustre-MDT0000: new disk, initializing [14952.235954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14952.265655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14957.565607] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14968.247743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14968.383887] Lustre: 505789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14968.432829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14968.437565] Lustre: Skipped 1 previous similar message [14968.524809] Lustre: lustre-MDT0001: new disk, initializing [14968.661748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14968.716282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14968.722653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14973.356483] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [14982.400744] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14988.046177] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14995.059424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14995.191506] Lustre: *** cfs_fail_loc=90f, val=0*** [14995.242189] Lustre: lustre-OST0000: new disk, initializing [14995.247519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14995.254550] Lustre: 507805:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14995.333467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15001.144532] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15002.646901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15002.651583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15002.712832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15011.067462] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15023.073978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15023.076771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15023.106809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15023.113401] Lustre: Skipped 1 previous similar message [15024.805915] Lustre: server umount lustre-OST0000 complete [15030.247400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15030.261833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15030.282867] Lustre: Skipped 1 previous similar message [15030.291071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15033.314234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15033.316524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15034.859397] Lustre: server umount lustre-MDT0000 complete [15038.692699] LustreError: 505780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628509 with bad export cookie 10839269932864630542 [15038.693635] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15038.699084] LustreError: 505780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15038.951663] Lustre: server umount lustre-MDT0001 complete [15053.764515] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [15056.216134] Key type lgssc unregistered [15056.592584] LNet: 509714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15056.604807] LNetError: 509714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15056.629554] LNet: Removed LNI 192.168.206.120@tcp [15057.595554] Key type .llcrypt unregistered [15057.597671] Key type ._llcrypt unregistered [15077.735347] Key type ._llcrypt registered [15077.737672] Key type .llcrypt registered [15077.889231] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [15078.650345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15078.694542] alg: No test for adler32 (adler32-zlib) [15079.768468] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [15080.141297] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [15081.791499] Key type lgssc registered [15083.090837] Lustre: Echo OBD driver; http://www.lustre.org/ [15091.464701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15091.482689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15092.848661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15096.896195] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15102.438945] LustreError: 510984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15102.461843] LustreError: 510984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15105.269776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15105.630491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15110.167432] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15117.246578] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15122.918553] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15129.471447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15129.801596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15135.927413] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15142.884777] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15154.070106] loop0: detected capacity change from 0 to 204800000 [15154.288537] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15154.302972] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15154.320545] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15160.511524] loop0: detected capacity change from 0 to 204800000 [15160.704527] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15160.712796] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15160.726160] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15165.387679] loop0: detected capacity change from 0 to 204800000 [15170.634366] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15170.796234] Lustre: fs15246-OST0000: new disk, initializing [15170.801157] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15170.807552] Lustre: 514579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15170.851685] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15176.890240] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15186.455461] loop1: detected capacity change from 0 to 204800000 [15191.132260] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15191.232733] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15191.297566] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15191.374630] Lustre: fs15246-MDT0000: new disk, initializing [15191.488527] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15191.507641] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15191.573845] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15192.610233] Lustre: 515701:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000eb7977e7 x1874464395540352/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787628679 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15195.072213] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15201.702579] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15201.719216] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15201.741468] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15216.621318] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15216.658885] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15221.735593] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15222.097833] Lustre: server umount fs15246-MDT0000 complete [15226.866372] LustreError: 513474:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15231.460879] Lustre: server umount fs15246-OST0000 complete [15242.209333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15242.210805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15242.218331] Lustre: Skipped 1 previous similar message [15247.360551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15247.374902] Lustre: Skipped 2 previous similar messages [15248.435357] Lustre: server umount lustre-OST0000 complete [15253.984495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15253.993612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15254.017314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15257.572328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15258.898636] Lustre: server umount lustre-MDT0000 complete [15262.690890] LustreError: 510984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15262.716399] LustreError: 510984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15263.027536] LustreError: 515695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628733 with bad export cookie 8307752606827084915 [15263.029986] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15263.048133] LustreError: 515695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15263.538371] Lustre: server umount lustre-MDT0001 complete [15277.927712] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [15280.947493] Key type lgssc unregistered [15281.263744] LNet: 518800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15281.270882] LNetError: 518800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15281.288980] LNet: Removed LNI 192.168.206.120@tcp [15282.497447] Key type .llcrypt unregistered [15282.501225] Key type ._llcrypt unregistered [15304.297502] Key type ._llcrypt registered [15304.300281] Key type .llcrypt registered [15304.419863] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [15305.723963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15305.765567] alg: No test for adler32 (adler32-zlib) [15306.847057] Lustre: Lustre: Build Version: 2.17.57_80_g74d5152 [15307.170103] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [15308.863523] Key type lgssc registered [15309.873275] Lustre: Echo OBD driver; http://www.lustre.org/ [15348.489663] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [15359.055332] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [15397.152052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15397.202694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15398.583579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15398.704074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15398.886537] Lustre: lustre-MDT0000: new disk, initializing [15399.013293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15399.061970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15403.863472] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15416.614576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15416.716458] Lustre: 523270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15416.767446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15416.770817] Lustre: Skipped 1 previous similar message [15416.852768] Lustre: lustre-MDT0001: new disk, initializing [15416.934837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15416.978954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15416.987266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15420.865873] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15430.057691] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15435.579575] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15442.552266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15442.808660] Lustre: lustre-OST0000: new disk, initializing [15442.813212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15442.822286] Lustre: 525237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15442.995364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15449.694555] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [15450.752736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15450.758282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15450.922286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15457.868724] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15461.613927] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15461.826545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15465.208321] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15465.359723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15471.584908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15471.587040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15471.601436] LustreError: Skipped 1 previous similar message [15471.603609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15471.615699] Lustre: Skipped 1 previous similar message [15473.335270] Lustre: server umount lustre-OST0000 complete [15478.241652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15478.256804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15478.274885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15478.284441] Lustre: Skipped 1 previous similar message [15481.830218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15481.830885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15486.946723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15486.957076] Lustre: Skipped 2 previous similar messages [15491.039177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15491.043760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15491.060628] Lustre: Skipped 1 previous similar message [15491.244283] Lustre: server umount lustre-MDT0000 complete [15494.761400] LustreError: 524064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787628965 with bad export cookie 5873774995480891926 [15494.767882] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15494.785089] LustreError: 524064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15495.016252] Lustre: server umount lustre-MDT0001 complete [15519.720615] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [15528.998659] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [15555.610182] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15225 sec ========= 23:37:05 (1787629025) [15557.304583] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:37:07 (1787629027) === [15561.118544] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:37:10 (1787629030) === [15603.874618] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [15607.120821] Key type lgssc unregistered [15607.389026] LNet: 531884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15607.408969] LNetError: 531884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15607.433511] LNet: Removed LNI 192.168.206.120@tcp [15608.052683] Key type .llcrypt unregistered [15608.056746] Key type ._llcrypt unregistered