[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487527800 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003107] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008453] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010025] pid_max: default: 32768 minimum: 301 [ 0.011160] LSM: Security Framework initializing [ 0.012066] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014098] *** VALIDATE selinux *** [ 0.023377] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028672] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032137] *** VALIDATE tmpfs *** [ 0.033504] *** VALIDATE proc *** [ 0.034273] *** VALIDATE cgroup *** [ 0.035000] *** VALIDATE cgroup2 *** [ 0.035283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036199] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042423] debug: unmapping init [mem 0xffffffffbc059000-0xffffffffbc060fff] [ 0.044952] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051024] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053301] rcu: Hierarchical SRCU implementation. [ 0.055437] smp: Bringing up secondary CPUs ... [ 0.056701] x86: Booting SMP configuration: [ 0.057039] .... node #0, CPUs: #1 #2 #3 [ 0.060588] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135488] node 0 deferred pages initialised in 69ms [ 0.138376] devtmpfs: initialized [ 0.140393] x86/mm: Memory block size: 128MB [ 0.142743] gcov: version magic: 0x41383552 [ 0.146215] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152327] pinctrl core: initialized pinctrl subsystem [ 0.154159] [ 0.154974] ************************************************************* [ 0.157012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159009] ** ** [ 0.161016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163013] ** ** [ 0.165016] ** This means that this kernel is built to expose internal ** [ 0.167014] ** IOMMU data structures, which may compromise security on ** [ 0.170018] ** your system. ** [ 0.173016] ** ** [ 0.175017] ** If you see this message and you are not debugging the ** [ 0.177018] ** kernel, report this immediately to your vendor! ** [ 0.180012] ** ** [ 0.182014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185015] ************************************************************* [ 0.187634] NET: Registered protocol family 16 [ 0.189485] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199084] cpuidle: using governor menu [ 0.200813] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203527] PCI: Using configuration type 1 for base access [ 0.205134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217087] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220249] cryptd: max_cpu_qlen set to 1000 [ 0.224291] ACPI: Added _OSI(Module Device) [ 0.226014] ACPI: Added _OSI(Processor Device) [ 0.227011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233265] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239438] ACPI: Interpreter enabled [ 0.242064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243014] ACPI: Using IOAPIC for interrupt routing [ 0.245119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249455] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258601] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270385] acpiphp: Slot [2] registered [ 0.272165] acpiphp: Slot [5] registered [ 0.274161] acpiphp: Slot [6] registered [ 0.276100] acpiphp: Slot [7] registered [ 0.277131] acpiphp: Slot [8] registered [ 0.279157] acpiphp: Slot [9] registered [ 0.280144] acpiphp: Slot [10] registered [ 0.281172] acpiphp: Slot [3] registered [ 0.283101] acpiphp: Slot [4] registered [ 0.284094] acpiphp: Slot [11] registered [ 0.285083] acpiphp: Slot [12] registered [ 0.286088] acpiphp: Slot [13] registered [ 0.287105] acpiphp: Slot [14] registered [ 0.288119] acpiphp: Slot [15] registered [ 0.289122] acpiphp: Slot [16] registered [ 0.291137] acpiphp: Slot [17] registered [ 0.292104] acpiphp: Slot [18] registered [ 0.294130] acpiphp: Slot [19] registered [ 0.296121] acpiphp: Slot [20] registered [ 0.297112] acpiphp: Slot [21] registered [ 0.299146] acpiphp: Slot [22] registered [ 0.300079] acpiphp: Slot [23] registered [ 0.301100] acpiphp: Slot [24] registered [ 0.303169] acpiphp: Slot [25] registered [ 0.305100] acpiphp: Slot [26] registered [ 0.307141] acpiphp: Slot [27] registered [ 0.308092] acpiphp: Slot [28] registered [ 0.310129] acpiphp: Slot [29] registered [ 0.311112] acpiphp: Slot [30] registered [ 0.313105] acpiphp: Slot [31] registered [ 0.315072] PCI host bridge to bus 0000:00 [ 0.316018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333296] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336396] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353790] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362497] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365907] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371888] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.389017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402731] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452975] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463313] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.486058] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.499609] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.510025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.549021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.565000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.571022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.578018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.601030] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.615273] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.624019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.631023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.647017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.660264] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.666015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.673017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.707022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.724305] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.726401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.728355] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.730399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.732287] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.737110] iommu: Default domain type: Passthrough [ 0.739507] SCSI subsystem initialized [ 0.741164] ACPI: bus type USB registered [ 0.742110] usbcore: registered new interface driver usbfs [ 0.744095] usbcore: registered new interface driver hub [ 0.746207] usbcore: registered new device driver usb [ 0.748176] pps_core: LinuxPPS API ver. 1 registered [ 0.750014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.753070] PTP clock support registered [ 0.756047] EDAC MC: Ver: 3.0.0 [ 0.758110] PCI: Using ACPI for IRQ routing [ 0.760959] NetLabel: Initializing [ 0.762013] NetLabel: domain hash size = 128 [ 0.764013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.766112] NetLabel: unlabeled traffic allowed by default [ 0.768138] vgaarb: loaded [ 0.769310] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.771020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.779684] clocksource: Switched to clocksource kvm-clock [ 0.891091] VFS: Disk quotas dquot_6.6.0 [ 0.892753] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.895856] *** VALIDATE ramfs *** [ 0.897204] *** VALIDATE hugetlbfs *** [ 0.898795] pnp: PnP ACPI init [ 0.901321] pnp: PnP ACPI: found 6 devices [ 0.931058] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.934876] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937249] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939580] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.942265] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.945176] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.948320] NET: Registered protocol family 2 [ 0.951126] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.956653] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.960652] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.966904] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.970853] TCP: Hash tables configured (established 65536 bind 65536) [ 0.974075] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.978286] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.981586] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.984763] NET: Registered protocol family 1 [ 0.988677] RPC: Registered named UNIX socket transport module. [ 0.991065] RPC: Registered udp transport module. [ 0.993076] RPC: Registered tcp transport module. [ 0.995124] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.998020] NET: Registered protocol family 44 [ 0.999762] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.002092] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.004357] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.007426] PCI: CLS 0 bytes, default 64 [ 1.009443] Unpacking initramfs... [ 2.436621] debug: unmapping init [mem 0xffffa031fcc54000-0xffffa031fffbffff] [ 2.442054] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.444678] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.448114] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.962904] Initialise system trusted keyrings [ 2.965016] Key type blacklist registered [ 2.967958] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.981270] zbud: loaded [ 2.984617] *** VALIDATE nfs *** [ 2.986064] *** VALIDATE nfs4 *** [ 2.988237] pstore: using deflate compression [ 2.992636] Platform Keyring initialized [ 3.114909] NET: Registered protocol family 38 [ 3.117096] Key type asymmetric registered [ 3.119232] Asymmetric key parser 'x509' registered [ 3.121155] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.125576] io scheduler mq-deadline registered [ 3.127695] io scheduler kyber registered [ 3.130698] io scheduler bfq registered [ 3.133326] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.136929] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.140243] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.143205] ACPI: Power Button [PWRF] [ 3.149412] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.157172] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.172769] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.182587] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.196383] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.225644] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.256803] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.264605] Non-volatile memory driver v1.3 [ 3.267374] Linux agpgart interface v0.103 [ 3.313515] virtio_blk virtio1: [vda] 136536 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.316909] vda: detected capacity change from 0 to 69906432 [ 3.336119] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.339981] vdb: detected capacity change from 0 to 1073741824 [ 3.357338] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.360220] vdc: detected capacity change from 0 to 2621440000 [ 3.375499] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.379065] vdd: detected capacity change from 0 to 2621440000 [ 3.395112] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.398662] vde: detected capacity change from 0 to 4294967296 [ 3.418539] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.421784] vdf: detected capacity change from 0 to 4294967296 [ 3.428646] libphy: Fixed MDIO Bus: probed [ 3.436979] usbcore: registered new interface driver usbserial_generic [ 3.441097] usbserial: USB Serial support registered for generic [ 3.444044] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.449115] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.451314] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.454167] mousedev: PS/2 mouse device common for all mice [ 3.457690] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.459926] rtc_cmos 00:05: RTC can wake from S4 [ 3.465299] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.465537] rtc_cmos 00:05: registered as rtc0 [ 3.470172] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.471325] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.476784] intel_pstate: CPU model not supported [ 3.479707] hid: raw HID events driver (C) Jiri Kosina [ 3.481711] usbcore: registered new interface driver usbhid [ 3.483360] usbhid: USB HID core driver [ 3.484802] drop_monitor: Initializing network drop monitor service [ 3.487326] Initializing XFRM netlink socket [ 3.489083] NET: Registered protocol family 10 [ 3.491904] Segment Routing with IPv6 [ 3.493503] NET: Registered protocol family 17 [ 3.495688] mpls_gso: MPLS GSO support [ 3.501352] RAS: Correctable Errors collector initialized. [ 3.503458] AVX version of gcm_enc/dec engaged. [ 3.504972] AES CTR mode by8 optimization enabled [ 3.584180] sched_clock: Marking stable (3584156522, 0)->(4523882070, -939725548) [ 3.588373] registered taskstats version 1 [ 3.590352] Loading compiled-in X.509 certificates [ 3.593182] zswap: loaded using pool lzo/zbud [ 3.617596] Key type big_key registered [ 3.630837] Key type encrypted registered [ 3.632423] ima: No TPM chip found, activating TPM-bypass! [ 3.634478] ima: Allocated hash algorithm: sha1 [ 3.636671] ima: No architecture policies found [ 3.639170] evm: Initialising EVM extended attributes: [ 3.641130] evm: security.selinux [ 3.642260] evm: security.ima [ 3.643563] evm: security.capability [ 3.644884] evm: HMAC attrs: 0x1 [ 3.648136] rtc_cmos 00:05: setting system clock to 2026-06-12 05:04:11 UTC (1781240651) [ 3.654459] debug: unmapping init [mem 0xffffffffbd003000-0xffffffffbd1fffff] [ 3.657411] debug: unmapping init [mem 0xffffffffbbd82000-0xffffffffbc058fff] [ 3.670141] Write protecting the kernel read-only data: 28672k [ 3.673641] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 3.676509] debug: unmapping init [mem 0xffffffffbad14000-0xffffffffbadfffff] [ 3.710227] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.718904] systemd[1]: Detected virtualization kvm. [ 3.720961] systemd[1]: Detected architecture x86-64. [ 3.723144] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.749339] systemd[1]: No hostname configured. [ 3.753293] systemd[1]: Set hostname to . [ 3.756426] random: systemd: uninitialized urandom read (16 bytes read) [ 3.759379] systemd[1]: Initializing machine ID from random generator. [ 3.892676] random: systemd: uninitialized urandom read (16 bytes read) [ 3.895615] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.900862] random: systemd: uninitialized urandom read (16 bytes read) [ 3.903748] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.909088] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.481685] device-mapper: uevent: version 1.0.3 [ 4.484133] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.173791] virtio_net virtio0 ens2: renamed from eth0 [ 5.185883] random: fast init done [ 5.224481] scsi host0: ata_piix [ 5.259967] scsi host1: ata_piix [ 5.265458] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.268899] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.924189] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.160853] random: crng init done [ 10.162405] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.470724] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.617592] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.898706] SELinux: Disabled at runtime. [ 11.963653] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.973153] systemd[1]: Detected virtualization kvm. [ 11.975406] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.502625] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.506217] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.513813] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.517399] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.520353] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.538751] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.543755] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 12.628298] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.940589] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.292548] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.358154] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.429137] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.444669] EDAC sbridge: Ver: 1.1.2 [ 14.786322] Key type dns_resolver registered [ 15.098127] NFS: Registering the id_resolver key type [ 15.100194] Key type id_resolver registered [ 15.101930] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg320-server login: [ 33.532024] hrtimer: interrupt took 10052058 ns [ 64.808678] libcfs: loading out-of-tree module taints kernel. [ 64.871716] Key type ._llcrypt registered [ 64.874812] Key type .llcrypt registered [ 65.035467] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 82.418643] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 84.244636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.257137] alg: No test for adler32 (adler32-zlib) [ 85.861675] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 86.869317] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 88.711160] Key type lgssc registered [ 90.632370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.064116] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 153.653449] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 167.982928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.020914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.374504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.422387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.531804] Lustre: lustre-MDT0000: new disk, initializing [ 169.641514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.673673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.361178] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.916286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.043822] Lustre: 6507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.123197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 188.132471] Lustre: Skipped 1 previous similar message [ 188.237040] Lustre: lustre-MDT0001: new disk, initializing [ 188.313458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 188.352400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 188.372083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.291351] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.367600] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.097014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.424643] Lustre: lustre-OST0000: new disk, initializing [ 208.429085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.435918] Lustre: 8409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.522226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.498524] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.115596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.123806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.226823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.555316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.686392] Lustre: lustre-OST0001: new disk, initializing [ 230.689379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 230.696877] Lustre: 9465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.789304] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.356843] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.129105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 239.141788] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 239.248711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.999697] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.666879] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.518385] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.401741] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing yml_node [ 276.173099] Lustre: DEBUG MARKER: Client: 2.17.53.67 [ 279.170859] Lustre: DEBUG MARKER: MDS: 2.17.53.67 [ 281.806488] Lustre: DEBUG MARKER: OSS: 2.17.53.67 [ 283.479977] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 01:08:48 EDT 2026 [ 303.879897] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 305.234868] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 321.503860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 321.509358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.531646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.114040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.117362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.155092] Lustre: Skipped 1 previous similar message [ 327.826942] Lustre: server umount lustre-MDT0000 complete [ 332.125912] LustreError: 7436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240979 with bad export cookie 2933901856113249822 [ 332.133353] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.150578] LustreError: 7436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 336.352393] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.365050] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.378391] Lustre: Skipped 1 previous similar message [ 336.410756] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 336.448691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.464732] Lustre: Skipped 4 previous similar messages [ 337.376617] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.412863] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 338.726649] Lustre: server umount lustre-MDT0001 complete [ 343.888659] Lustre: server umount lustre-OST0000 complete [ 347.959620] Lustre: server umount lustre-OST0001 complete [ 355.053647] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 364.205026] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 405.713489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.176918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.233968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.298636] Lustre: lustre-MDT0000: new disk, initializing [ 406.469265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.522854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.035764] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 424.703050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.886653] Lustre: 14984:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 425.037586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 425.041540] Lustre: Skipped 1 previous similar message [ 425.195366] Lustre: lustre-MDT0001: new disk, initializing [ 425.333071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 425.398662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 425.415220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 430.641596] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 441.126309] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 446.859869] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 455.338183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.785975] Lustre: lustre-OST0000: new disk, initializing [ 455.795791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.802984] Lustre: 16728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.938639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.686593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.702484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 457.862313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 463.675830] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 472.771424] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.934807] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 478.263278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 483.660725] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 483.942520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.426731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.434916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.448526] Lustre: Skipped 1 previous similar message [ 488.457207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.461246] LustreError: Skipped 1 previous similar message [ 488.483043] Lustre: Skipped 3 previous similar messages [ 492.153468] Lustre: server umount lustre-OST0000 complete [ 497.121066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 497.138543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.156761] Lustre: Skipped 1 previous similar message [ 497.166164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 502.463617] Lustre: server umount lustre-MDT0000 complete [ 503.778486] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.812755] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 507.374203] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241155 with bad export cookie 2933901856113251355 [ 507.375903] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.389501] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 507.774505] Lustre: server umount lustre-MDT0001 complete [ 515.122561] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:12:40 (1781241160) [ 538.437856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.082825] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.208818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.230579] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.683768] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 553.452067] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.476468] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 555.247257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.780707] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 567.164378] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 571.884463] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 574.520267] Lustre: Failing over lustre-MDT0000 [ 574.934848] Lustre: server umount lustre-MDT0000 complete [ 575.459630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.470607] Lustre: Skipped 1 previous similar message [ 575.474827] LustreError: 19452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.486952] LustreError: 19452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 591.776106] Lustre: 3657:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781241223/real 1781241223] req@ffffa031438e3480 x1867766284058112/t0(0) o400->MGC192.168.203.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781241239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 591.823793] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.919267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.080399] LustreError: 3653:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0326f288780 x1867766284061056/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 602.598644] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 607.728044] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 607.753759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 607.789818] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 608.488971] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 617.378677] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 621.682841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.701253] Lustre: Skipped 2 previous similar messages [ 621.708325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 621.716586] Lustre: Skipped 3 previous similar messages [ 628.054807] Lustre: server umount lustre-MDT0000 complete [ 631.277101] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.310644] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 632.425671] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241280 with bad export cookie 2933901856113252769 [ 632.433763] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.441968] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 632.777720] Lustre: server umount lustre-MDT0001 complete [ 642.265402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.902274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.914113] Lustre: Skipped 2 previous similar messages [ 648.138065] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 656.605506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.318187] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 667.440475] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 671.874950] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 674.058374] Lustre: Failing over lustre-MDT0000 [ 674.319773] Lustre: server umount lustre-MDT0000 complete [ 677.857421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 677.862674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.889419] Lustre: Skipped 1 previous similar message [ 693.285205] Lustre: 3656:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781241325/real 1781241325] req@ffffa03252273840 x1867766284087296/t0(0) o400->MGC192.168.203.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781241341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 693.325430] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.800161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.848291] LustreError: 22452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 698.875601] LustreError: 22452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 703.968041] LustreError: 3653:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa03142a2b0c0 x1867766284090368/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 704.395887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 709.115505] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 709.605573] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 709.626390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 709.640061] Lustre: Skipped 1 previous similar message [ 709.687943] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 718.140475] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 724.980791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.990401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.030798] Lustre: Skipped 3 previous similar messages [ 729.172779] Lustre: server umount lustre-MDT0000 complete [ 733.544059] LustreError: 22436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241381 with bad export cookie 2933901856113253707 [ 733.553820] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.554990] LustreError: 22436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 733.923018] Lustre: server umount lustre-MDT0001 complete [ 746.044809] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:16:30 (1781241390) [ 764.448398] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 774.092674] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 812.123542] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 824.024346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.310903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 824.352558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 824.422368] Lustre: lustre-MDT0000: new disk, initializing [ 824.502442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.506665] Lustre: Skipped 2 previous similar messages [ 824.516791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 829.715886] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 841.480050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.771166] Lustre: 28444:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 842.385186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 847.479154] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 856.131169] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 860.619905] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 868.909108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 869.236795] Lustre: 30182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 871.184375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 871.248445] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 871.337405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 871.359397] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 876.543396] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 883.923891] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 889.562476] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 912.355910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 912.357361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.364967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.375242] Lustre: Skipped 3 previous similar messages [ 912.390751] Lustre: Skipped 3 previous similar messages [ 915.873983] Lustre: server umount lustre-OST0000 complete [ 926.021765] Lustre: server umount lustre-MDT0000 complete [ 927.715308] LustreError: 29280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.732334] LustreError: 29280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 930.182670] LustreError: 28434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241578 with bad export cookie 2933901856113254113 [ 930.190347] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 930.202826] LustreError: 28434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 930.704586] Lustre: server umount lustre-MDT0001 complete [ 941.912904] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 945.086238] Key type lgssc unregistered [ 945.520192] LNet: 32032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.530449] LNetError: 32032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.546318] LNet: Removed LNI 192.168.203.120@tcp [ 946.395149] Key type .llcrypt unregistered [ 946.396957] Key type ._llcrypt unregistered [ 970.467431] Key type ._llcrypt registered [ 970.470240] Key type .llcrypt registered [ 970.614691] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 971.686348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 971.705402] alg: No test for adler32 (adler32-zlib) [ 972.890878] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 973.086339] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 974.775156] Key type lgssc registered [ 975.847597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 983.841995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 983.866929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.310434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.583948] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 995.817255] LustreError: 33429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.841920] LustreError: 33429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 998.233614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.486220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.175347] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1008.861116] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1013.003620] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1020.327304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1020.681125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.800080] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1026.802689] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1031.024644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1033.459104] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1049.973272] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:21:34 (1781241694) [ 1052.476368] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1054.477982] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1062.228697] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:21:47 (1781241707) [ 1071.281310] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:21:56 (1781241716) [ 1072.584233] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1081.894192] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1081.904506] Lustre: Skipped 2 previous similar messages [ 1084.072814] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:22:09 (1781241729) [ 1086.725989] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1086.734361] Lustre: Skipped 2 previous similar messages [ 1096.745939] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1096.762718] Lustre: Skipped 4 previous similar messages [ 1112.785179] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:22:38 (1781241758) [ 1115.444169] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1118.793973] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1127.338320] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:22:53 (1781241773) [ 1129.101884] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1129.106594] Lustre: Skipped 6 previous similar messages [ 1141.579944] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:23:06 (1781241786) [ 1157.090880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.110413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.137408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.138781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1159.152664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.154608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.173738] Lustre: Skipped 1 previous similar message [ 1164.256645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.267616] Lustre: Skipped 1 previous similar message [ 1168.864202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1169.014608] Lustre: server umount lustre-OST0000 complete [ 1174.517691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.532108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.551964] Lustre: Skipped 1 previous similar message [ 1174.570365] Lustre: Skipped 1 previous similar message [ 1179.625062] LustreError: 33429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.660868] LustreError: 33429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1179.771655] Lustre: server umount lustre-MDT0000 complete [ 1183.763610] LustreError: 33409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241831 with bad export cookie 15481479686254908509 [ 1183.765121] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.770902] LustreError: 33409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1184.259036] Lustre: server umount lustre-MDT0001 complete [ 1194.608090] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1197.795984] Key type lgssc unregistered [ 1198.156870] LNet: 40928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.162165] LNetError: 40928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.176902] LNet: Removed LNI 192.168.203.120@tcp [ 1198.985439] Key type .llcrypt unregistered [ 1198.987994] Key type ._llcrypt unregistered [ 1215.825555] Key type ._llcrypt registered [ 1215.828338] Key type .llcrypt registered [ 1215.966164] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 1230.697530] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1231.963743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.136169] alg: No test for adler32 (adler32-zlib) [ 1233.254558] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 1233.495541] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1235.135234] Key type lgssc registered [ 1236.585194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1272.271513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.288654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.507547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1273.534793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1273.593371] Lustre: lustre-MDT0000: new disk, initializing [ 1273.685590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.709499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1278.017274] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1288.206811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.325550] Lustre: 44048:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1288.395433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1288.399833] Lustre: Skipped 1 previous similar message [ 1288.471432] Lustre: lustre-MDT0001: new disk, initializing [ 1288.533767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1288.572492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1288.581085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1292.970075] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1299.991591] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1303.692255] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1310.252111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.489833] Lustre: lustre-OST0000: new disk, initializing [ 1310.495375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1310.501613] Lustre: 45789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1310.599965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.150528] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1318.497108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1318.516222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1318.635058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1325.747544] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1331.444259] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1331.790809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1336.942019] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1337.182462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1344.484365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1344.496193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.499242] LustreError: Skipped 1 previous similar message [ 1344.505373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1344.510340] Lustre: Skipped 1 previous similar message [ 1345.930380] Lustre: server umount lustre-OST0000 complete [ 1354.720249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.727256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.738848] Lustre: Skipped 1 previous similar message [ 1354.745794] Lustre: Skipped 2 previous similar messages [ 1356.502172] Lustre: server umount lustre-MDT0000 complete [ 1360.066719] LustreError: 44040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242007 with bad export cookie 11978398272276685683 [ 1360.070997] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1360.076063] LustreError: 44040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1360.372869] Lustre: server umount lustre-MDT0001 complete [ 1369.021719] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:26:54 (1781242014) [ 1375.088047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.599518] LustreError: 48104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.623875] LustreError: 48104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1375.698604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.361444] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1380.832757] LustreError: 48105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.952270] LustreError: 48104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.093809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.507121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.861598] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1399.659659] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1403.634896] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1410.624459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.131546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.372315] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1424.012418] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1428.936639] LustreError: 50292:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1430.042470] Lustre: Modifying parameter general.timeout=1 in log params [ 1431.388642] Lustre: Modifying parameter general.timeout=2 in log params [ 1433.734035] Lustre: Modifying parameter general.timeout=4 in log params [ 1433.742084] Lustre: Skipped 1 previous similar message [ 1437.769282] Lustre: Modifying parameter general.timeout=8 in log params [ 1437.772707] Lustre: Skipped 3 previous similar messages [ 1446.731299] Lustre: Modifying parameter general.timeout=17 in log params [ 1446.754471] Lustre: Skipped 8 previous similar messages [ 1463.285238] Lustre: Modifying parameter general.timeout=34 in log params [ 1463.288544] Lustre: Skipped 16 previous similar messages [ 1495.402267] Lustre: Modifying parameter general.timeout=67 in log params [ 1495.406455] Lustre: Skipped 32 previous similar messages [ 1561.398939] Lustre: Disabling parameter general.timeout= in log params [ 1561.402233] Lustre: Skipped 62 previous similar messages [ 1564.535303] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:30:09 (1781242209) [ 1567.476160] Lustre: 57242:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1569.446144] Lustre: 57340:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1573.176966] Lustre: 57550:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1573.184843] Lustre: 57550:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1577.370325] Lustre: 57746:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1577.390325] Lustre: 57746:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1604.645080] Lustre: 59113:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1604.663149] LustreError: 59113:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1609.537636] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:30:54 (1781242254) [ 1625.057745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1625.061858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.069451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.130512] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.135976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.166135] Lustre: Skipped 2 previous similar messages [ 1630.854666] Lustre: server umount lustre-MDT0000 complete [ 1633.248764] LustreError: 48104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.281843] LustreError: 48104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1634.725430] LustreError: 48085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242282 with bad export cookie 11978398272276686565 [ 1634.732200] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1634.741735] LustreError: 48085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1635.008146] Lustre: server umount lustre-MDT0001 complete [ 1648.397693] Lustre: server umount lustre-OST0000 complete [ 1678.971656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.344872] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1679.593790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.563784] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1695.790675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.019474] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1696.095156] Lustre: 61649:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1703.724113] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1710.015319] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.460630] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1722.152740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1722.378920] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1722.662802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.669144] Lustre: Skipped 1 previous similar message [ 1728.635931] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1735.201394] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1737.025781] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1737.045146] Lustre: Skipped 23 previous similar messages [ 1737.099177] Lustre: 63876:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1747.425874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.446592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.467366] Lustre: Skipped 1 previous similar message [ 1747.476246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1750.071628] Lustre: server umount lustre-OST0000 complete [ 1758.188927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1758.192243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.199327] LustreError: Skipped 1 previous similar message [ 1758.231324] Lustre: Skipped 1 previous similar message [ 1758.235668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.239537] Lustre: Skipped 1 previous similar message [ 1760.400420] Lustre: server umount lustre-MDT0000 complete [ 1763.299761] LustreError: 62882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1763.339936] LustreError: 62882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1764.551204] LustreError: 61640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242412 with bad export cookie 11978398272276689638 [ 1764.555945] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.561787] LustreError: 61640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1765.057323] Lustre: server umount lustre-MDT0001 complete [ 1775.146232] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1777.845563] Key type lgssc unregistered [ 1778.233782] LNet: 65070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.247645] LNetError: 65070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.266114] LNet: Removed LNI 192.168.203.120@tcp [ 1779.086735] Key type .llcrypt unregistered [ 1779.091567] Key type ._llcrypt unregistered [ 1803.910812] Key type ._llcrypt registered [ 1803.913820] Key type .llcrypt registered [ 1804.062844] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1805.237598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.380803] alg: No test for adler32 (adler32-zlib) [ 1806.401542] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 1806.663540] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1808.361163] Key type lgssc registered [ 1809.642040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1817.882778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1817.901759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.657790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.076875] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1829.858839] LustreError: 66466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1829.880591] LustreError: 66466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1833.361379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.313452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1839.462110] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1845.741887] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1849.337134] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1855.990561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1856.590624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.595863] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1868.213882] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1875.639757] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1885.154400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1885.163389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.176625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.666409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.677093] Lustre: Skipped 1 previous similar message [ 1885.679695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.684858] Lustre: Skipped 1 previous similar message [ 1889.423707] Lustre: server umount lustre-MDT0000 complete [ 1890.798100] LustreError: 68625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.818300] LustreError: 68625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1893.587956] LustreError: 66448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242541 with bad export cookie 1912286359540038697 [ 1893.593493] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.610499] LustreError: 66448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1894.010064] Lustre: server umount lustre-MDT0001 complete [ 1908.394927] Lustre: server umount lustre-OST0000 complete [ 1934.650783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.073623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1935.113526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1935.375294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.996108] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1948.727071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.812101] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1948.837461] Lustre: 70995:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1949.175561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.693218] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1960.383041] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1964.917650] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1972.250989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1972.592036] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1973.142979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.847085] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1987.137688] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1989.403050] Lustre: 73233:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1991.346522] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1997.798343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1997.802737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.808385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1997.810639] LustreError: Skipped 1 previous similar message [ 1997.839085] Lustre: Skipped 1 previous similar message [ 2002.269736] Lustre: server umount lustre-OST0000 complete [ 2008.034188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.046576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.047576] Lustre: Skipped 1 previous similar message [ 2008.058549] Lustre: Skipped 1 previous similar message [ 2012.582173] Lustre: server umount lustre-MDT0000 complete [ 2013.159025] LustreError: 71007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.179917] LustreError: 71007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2016.462200] LustreError: 70987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242664 with bad export cookie 1912286359540039530 [ 2016.475807] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.477984] LustreError: 70987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2016.867686] Lustre: server umount lustre-MDT0001 complete [ 2028.266606] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 2031.020274] Key type lgssc unregistered [ 2031.318096] LNet: 74429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.322909] LNetError: 74429:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.336242] LNet: Removed LNI 192.168.203.120@tcp [ 2032.186889] Key type .llcrypt unregistered [ 2032.197227] Key type ._llcrypt unregistered [ 2056.673884] Key type ._llcrypt registered [ 2056.677846] Key type .llcrypt registered [ 2056.815843] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2057.955711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.228449] alg: No test for adler32 (adler32-zlib) [ 2059.372443] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 2059.618118] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 2061.287161] Key type lgssc registered [ 2062.700710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2070.910163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2070.924786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.367740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.264864] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2082.787146] LustreError: 75826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.811810] LustreError: 75826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2085.370293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.105089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.749799] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2096.532703] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2100.429809] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2107.126312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2107.794531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.125777] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2120.338223] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2126.677678] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2127.603055] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2129.358627] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2129.363956] Lustre: Skipped 1 previous similar message [ 2132.294583] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2132.301676] Lustre: Skipped 2 previous similar messages [ 2136.871356] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2136.877892] Lustre: Skipped 4 previous similar messages [ 2145.068582] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2145.071256] Lustre: Skipped 7 previous similar messages [ 2161.767494] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2161.773890] Lustre: Skipped 17 previous similar messages [ 2194.058373] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2194.064614] Lustre: Skipped 34 previous similar messages [ 2228.201416] Lustre: 83585:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2231.055584] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:41:16 (1781242876) [ 2232.714900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2234.585433] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:41:19 (1781242879) [ 2238.951575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2238.957720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.968540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2240.992704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.999172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.003333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.035590] Lustre: Skipped 1 previous similar message [ 2244.401835] Lustre: server umount lustre-OST0000 complete [ 2249.696145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2249.706906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.719617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.676488] Lustre: server umount lustre-MDT0000 complete [ 2256.355452] LustreError: 76540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.372695] LustreError: 76540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2258.429274] LustreError: 77450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242906 with bad export cookie 6473317774288727133 [ 2258.440462] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.446504] LustreError: 77450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2258.751518] Lustre: server umount lustre-MDT0001 complete [ 2268.709474] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 2270.949956] Key type lgssc unregistered [ 2271.220205] LNet: 84821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.230622] LNetError: 84821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.243579] LNet: Removed LNI 192.168.203.120@tcp [ 2272.095634] Key type .llcrypt unregistered [ 2272.099231] Key type ._llcrypt unregistered [ 2275.878596] Key type ._llcrypt registered [ 2275.880893] Key type .llcrypt registered [ 2275.998422] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2279.602786] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules [ 2280.409160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.430157] alg: No test for adler32 (adler32-zlib) [ 2281.587315] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 2281.782698] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 2283.474174] Key type lgssc registered [ 2284.577512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2285.743810] LustreError: 85988:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2301.942195] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2307.900120] LustreError: 85988:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2309.615413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2309.628660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.045108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.145780] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2320.353820] LustreError: 86994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.384354] LustreError: 86994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2323.551956] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:42:48 (1781242968) [ 2325.490970] LustreError: 86995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.113512] LustreError: 87026:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2328.124835] LustreError: 87026:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2328.131509] LustreError: 87026:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2328.504150] Lustre: server umount lustre-MDT0000 complete [ 2341.314217] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 2343.876606] Key type lgssc unregistered [ 2344.164311] LNet: 88485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.178404] LNetError: 88485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.200142] LNet: Removed LNI 192.168.203.120@tcp [ 2345.109897] Key type .llcrypt unregistered [ 2345.115145] Key type ._llcrypt unregistered [ 2360.610923] Key type ._llcrypt registered [ 2360.613314] Key type .llcrypt registered [ 2360.698686] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2361.785197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2361.870637] alg: No test for adler32 (adler32-zlib) [ 2362.866507] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 2363.112224] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 2364.823267] Key type lgssc registered [ 2366.049306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2373.464114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2373.484906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.060406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.885868] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2384.352398] LustreError: 89685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.365715] LustreError: 89685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2387.135330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.702317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.634592] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2396.602619] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2399.594447] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2405.601496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2406.187395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.951616] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2417.232363] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2464.332760] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:45:09 (1781243109) [ 2465.827062] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2467.395465] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:45:13 (1781243113) [ 2470.887628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.891348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2470.898962] Lustre: Skipped 2 previous similar messages [ 2470.911656] Lustre: Skipped 2 previous similar messages [ 2476.002132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.010657] Lustre: Skipped 2 previous similar messages [ 2476.854430] Lustre: server umount lustre-MDT0000 complete [ 2479.011989] LustreError: 90400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.026401] LustreError: 90400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2479.925224] LustreError: 89666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243127 with bad export cookie 2724302060749796109 [ 2479.940955] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.191603] Lustre: server umount lustre-MDT0001 complete [ 2493.781822] Lustre: server umount lustre-OST0000 complete [ 2503.705605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.084768] LustreError: 93622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.210048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.589800] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2509.281186] LustreError: 93623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.384425] LustreError: 93622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.050321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.632510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2519.220778] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2524.033589] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2527.379769] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2537.182222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2537.328923] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2537.333938] LustreError: 93611:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2537.357708] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2537.367201] LustreError: 95481:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2537.374199] LustreError: 95481:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2537.378129] LustreError: 95481:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2537.419716] Lustre: server umount lustre-OST0000 complete [ 2537.422286] LustreError: 95481:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2543.736284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2543.845820] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2543.851236] LustreError: 93612:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2543.857301] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2543.863259] LustreError: 95941:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2543.869253] LustreError: 95941:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2543.878493] LustreError: 95941:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2543.922733] Lustre: server umount lustre-OST0000 complete [ 2543.928997] LustreError: 95941:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2552.056940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2552.269509] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2552.274816] Lustre: Found index 0 for lustre-OST0000, updating log [ 2552.278515] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2552.341406] Lustre: lustre-OST0000: new disk, initializing [ 2552.346945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2552.356424] Lustre: 96474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2552.569644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.308706] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2561.862696] Lustre: Failing over lustre-OST0000 [ 2562.060818] Lustre: server umount lustre-OST0000 complete [ 2565.903634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.909955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.914254] Lustre: Skipped 1 previous similar message [ 2565.944616] Lustre: Skipped 1 previous similar message [ 2571.233467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2571.244446] Lustre: Skipped 1 previous similar message [ 2572.415720] Lustre: server umount lustre-MDT0000 complete [ 2575.979252] LustreError: 94346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243223 with bad export cookie 2724302060749798853 [ 2575.982336] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.985530] LustreError: 94346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2585.751631] Lustre: server umount lustre-MDT0001 complete [ 2594.091374] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:47:19 (1781243239) [ 2607.439830] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2615.617619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.931776] LustreError: 98785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.030384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2619.767344] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2627.012682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.315441] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2633.694655] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2639.122745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.692060] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2640.744061] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2640.755539] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2640.769684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2644.931376] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2645.984468] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2645.999136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2651.104814] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 2651.116929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2652.575474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.671564] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2652.732645] Lustre: lustre-OST0001: new disk, initializing [ 2652.737577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2652.742365] Lustre: 101051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2652.953948] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2652.960112] Lustre: Skipped 2 previous similar messages [ 2653.986282] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:46 [ 2654.006304] Lustre: Skipped 1 previous similar message [ 2654.017355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2654.027927] LustreError: Skipped 1 previous similar message [ 2654.054099] Lustre: lustre-OST0000: Recovery over after 0:14, of 1 clients 1 recovered and 0 were evicted. [ 2654.054479] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2657.967274] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2659.856233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2659.863415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2659.952380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2664.749469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2670.404588] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2673.633135] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2673.640856] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.653974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2679.777784] Lustre: server umount lustre-MDT0001 complete [ 2680.295020] LustreError: 98780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.324632] LustreError: 98780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2687.639659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.708530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2687.713480] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2687.749197] Lustre: 98773:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2689.780164] LustreError: 102654:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2689.786230] LustreError: 102654:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2689.792797] LustreError: 98777:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2689.858978] LustreError: 102662:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.120@tcp: cfg command failed: rc = -17 [ 2689.863819] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2689.863819] [ 2691.334879] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2693.612454] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2698.966972] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:49:04 (1781243344) [ 2713.715857] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2729.332913] Lustre: 104615:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2748.730642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2751.397548] Lustre: 105748:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2826.209623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2826.219927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.228366] Lustre: Skipped 2 previous similar messages [ 2826.232722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.236093] Lustre: Skipped 4 previous similar messages [ 2831.838889] Lustre: server umount lustre-MDT0000 complete [ 2831.842578] LustreError: 100335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.864719] LustreError: 100335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2834.518881] LustreError: 98766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243482 with bad export cookie 2724302060749799784 [ 2834.522884] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.525329] LustreError: 98766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2834.701871] Lustre: server umount lustre-MDT0001 complete [ 2847.611774] Lustre: server umount lustre-OST0000 complete [ 2860.552860] Lustre: server umount lustre-OST0001 complete [ 2864.760853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2871.327496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2876.856639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2882.808443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2895.748970] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2902.292844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.310167] Lustre: lustre-MDT0000: reset Object Index mappings [ 2902.630927] LustreError: 109680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.703862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.709656] Lustre: Skipped 1 previous similar message [ 2905.595813] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2911.179290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.195350] Lustre: lustre-MDT0001: reset Object Index mappings [ 2914.665867] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2916.709772] Lustre: 110793:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2921.527193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2926.324578] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2927.088024] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 2928.111249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2932.239219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.688465] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2937.845178] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2937.846497] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2942.107253] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2944.581406] Lustre: 112634:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2979.284789] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:53:45 (1781243625) [ 3008.992304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3008.996581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.005318] Lustre: Skipped 3 previous similar messages [ 3009.010367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.015226] Lustre: Skipped 3 previous similar messages [ 3021.279264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3021.419779] Lustre: server umount lustre-MDT0000 complete [ 3023.717556] LustreError: 109661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243671 with bad export cookie 2724302060749846677 [ 3023.723577] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.730791] LustreError: 109661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3024.867495] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.891404] Lustre: Skipped 4 previous similar messages [ 3034.016753] LustreError: 112656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3034.031184] LustreError: 112656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3038.175181] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3038.360443] Lustre: server umount lustre-MDT0001 complete [ 3055.071166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3055.167914] Lustre: server umount lustre-OST0000 complete [ 3063.602388] Lustre: server umount lustre-OST0001 complete [ 3068.676523] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3082.968967] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3135.219268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.683403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3135.722583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3135.785993] Lustre: lustre-MDT0000: new disk, initializing [ 3135.863452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.870429] Lustre: Skipped 3 previous similar messages [ 3135.884499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3140.166149] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3147.277525] Lustre: server umount lustre-MDT0000 complete [ 3157.378346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.821603] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3157.831122] Lustre: 116855:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3157.842524] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3163.327502] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3168.221554] Lustre: server umount lustre-MDT0000 complete [ 3176.667445] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:57:01 (1781243821) [ 3177.964957] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3179.693180] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:57:05 (1781243825) [ 3196.705780] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3206.196863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.703464] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3219.280667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.423474] Lustre: 118873:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3219.514577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3219.526989] Lustre: Skipped 1 previous similar message [ 3219.628583] Lustre: lustre-MDT0001: new disk, initializing [ 3219.817501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3219.828167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3224.326611] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3229.175290] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3235.829556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.034813] Lustre: lustre-OST0000: new disk, initializing [ 3236.040374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3236.048860] Lustre: 120416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3237.670717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3237.681972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3237.827631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3241.261551] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3251.727612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3251.872838] Lustre: lustre-OST0001: new disk, initializing [ 3251.877343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3251.882940] Lustre: 121269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3253.624029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3253.633950] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3253.745880] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3257.978235] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3267.185425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3270.693847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3694.260794] Lustre: 127687:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3694.276281] Lustre: 127687:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3694.285507] Lustre: 127687:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3694.299577] Lustre: 127687:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3694.321557] Lustre: 127687:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3694.330234] Lustre: 127687:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3985.953455] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:10:30 (1781244630) [ 4001.258089] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.262869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.291826] Lustre: Skipped 3 previous similar messages [ 4001.293805] Lustre: Skipped 20 previous similar messages [ 4006.376957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.380895] Lustre: Skipped 3 previous similar messages [ 4007.331393] Lustre: server umount lustre-MDT0000 complete [ 4011.460361] LustreError: 127701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244659 with bad export cookie 2724302060749910468 [ 4011.461099] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4011.468262] LustreError: 127701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4011.489112] LustreError: 121669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.501421] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.506562] LustreError: 121669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4011.531371] Lustre: Skipped 1 previous similar message [ 4016.610357] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4016.627889] Lustre: Skipped 3 previous similar messages [ 4018.068482] Lustre: server umount lustre-MDT0001 complete [ 4028.385862] Lustre: server umount lustre-OST0000 complete [ 4038.597554] Lustre: server umount lustre-OST0001 complete [ 4046.381106] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4057.678963] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4099.521436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.877137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4099.934235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4100.016607] Lustre: lustre-MDT0000: new disk, initializing [ 4100.110834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.114405] Lustre: Skipped 5 previous similar messages [ 4100.129032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4105.292779] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4118.124732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.224734] Lustre: 130722:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4118.326728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4118.342192] Lustre: Skipped 1 previous similar message [ 4118.480917] Lustre: lustre-MDT0001: new disk, initializing [ 4118.784226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4118.816762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4125.247778] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4134.093245] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4138.409192] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4144.936016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4145.295983] Lustre: lustre-OST0000: new disk, initializing [ 4145.300281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4145.306495] Lustre: 132466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4145.434112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.438234] Lustre: Skipped 1 previous similar message [ 4147.281656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4147.297122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4147.405838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4152.169485] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4160.603211] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4172.889281] Lustre: lustre-MDD0000: changelog on [ 4175.636236] Lustre: lustre-MDD0001: changelog on [ 6324.389985] Lustre: lustre-MDD0001: changelog off [ 6326.400338] Lustre: 130727:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6327.255806] Lustre: lustre-MDD0000: changelog off [ 6333.930445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6333.930923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.944044] LustreError: Skipped 1 previous similar message [ 6333.963190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6338.863381] Lustre: server umount lustre-OST0000 complete [ 6345.189658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6345.194353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6345.206115] Lustre: Skipped 2 previous similar messages [ 6345.229073] Lustre: Skipped 4 previous similar messages [ 6348.970907] Lustre: server umount lustre-MDT0000 complete [ 6350.314281] LustreError: 131542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.341454] LustreError: 131542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6353.732988] LustreError: 132465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247001 with bad export cookie 2724302060750691899 [ 6353.740303] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6353.759947] LustreError: 132465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6354.553556] Lustre: server umount lustre-MDT0001 complete [ 6370.547718] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 6373.246367] Key type lgssc unregistered [ 6373.495886] LNet: 141502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6373.506547] LNetError: 141502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6373.538294] LNet: Removed LNI 192.168.203.120@tcp [ 6374.268349] Key type .llcrypt unregistered [ 6374.276353] Key type ._llcrypt unregistered [ 6394.882982] Key type ._llcrypt registered [ 6394.885448] Key type .llcrypt registered [ 6395.007672] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 6417.808627] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6419.819405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6419.895688] alg: No test for adler32 (adler32-zlib) [ 6421.781381] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 6422.155337] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 6423.887142] Key type lgssc registered [ 6426.260796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6473.080247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6473.126491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6474.492202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6474.534663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6474.612833] Lustre: lustre-MDT0000: new disk, initializing [ 6474.731663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6474.767670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6479.531681] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6491.091248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.238238] Lustre: 144674:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6491.304192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6491.311796] Lustre: Skipped 1 previous similar message [ 6491.373944] Lustre: lustre-MDT0001: new disk, initializing [ 6491.442620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6491.485596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6491.497190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6495.681969] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6502.835461] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6506.075398] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6513.681186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6514.136929] Lustre: lustre-OST0000: new disk, initializing [ 6514.145452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6514.161894] Lustre: 146416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6514.309530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6520.760525] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6523.948100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6523.956157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6524.129512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6528.972867] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6534.261611] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_module obdecho/obdecho [ 6536.326350] LustreError: 147292:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6536.335758] LustreError: 147292:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6537.837914] LustreError: 147388:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6553.058776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6553.084896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.128541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.232323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.232801] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.257702] Lustre: Skipped 1 previous similar message [ 6555.268487] Lustre: Skipped 1 previous similar message [ 6559.355674] Lustre: server umount lustre-MDT0000 complete [ 6563.032490] LustreError: 145464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247210 with bad export cookie 7086422915484077988 [ 6563.040703] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6563.050180] LustreError: 145464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6563.436308] Lustre: server umount lustre-MDT0001 complete [ 6577.889741] Lustre: server umount lustre-OST0000 complete [ 6600.904220] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 6611.579356] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 6650.681911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.092961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6651.120663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6651.199919] Lustre: lustre-MDT0000: new disk, initializing [ 6651.346191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.364849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6656.272649] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6670.594078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.695846] Lustre: 151069:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6670.742518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6670.755664] Lustre: Skipped 1 previous similar message [ 6670.828814] Lustre: lustre-MDT0001: new disk, initializing [ 6670.977171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6671.060050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6671.089436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6676.011912] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6683.174612] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6686.693244] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6693.266461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6693.490036] Lustre: lustre-OST0000: new disk, initializing [ 6693.495040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6693.510132] Lustre: 152814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6693.582287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.952898] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6700.594942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6700.603490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6700.712348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6705.850053] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6709.931302] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6710.154257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6713.986492] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6714.174446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6716.387270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6716.387403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.396982] LustreError: Skipped 1 previous similar message [ 6716.398530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6716.398535] Lustre: Skipped 1 previous similar message [ 6716.465430] Lustre: Skipped 1 previous similar message [ 6721.507509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6721.510769] Lustre: Skipped 2 previous similar messages [ 6726.626739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.639281] Lustre: Skipped 1 previous similar message [ 6730.207444] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6730.319265] Lustre: server umount lustre-OST0000 complete [ 6736.866070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.867768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6736.892899] Lustre: Skipped 1 previous similar message [ 6736.909332] Lustre: Skipped 1 previous similar message [ 6739.789574] Lustre: server umount lustre-MDT0000 complete [ 6741.985408] LustreError: 152199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.002680] LustreError: 152199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6743.107473] LustreError: 151063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247390 with bad export cookie 7086422915484078954 [ 6743.111127] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6743.121828] LustreError: 151063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6743.473804] Lustre: server umount lustre-MDT0001 complete [ 6746.487511] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:56:32 (1781247392) [ 6747.946789] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6749.366231] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:56:35 (1781247395) [ 6756.507904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.947260] LustreError: 155078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.041882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6760.985334] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6762.464849] LustreError: 155078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.489399] LustreError: 155078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6766.567898] LustreError: 155097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.137268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6773.666501] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6778.757521] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6781.819841] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6787.363797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6793.259103] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6798.650234] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6810.592299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6810.598689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.611528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6816.134931] Lustre: server umount lustre-MDT0000 complete [ 6817.760175] LustreError: 155079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.781477] LustreError: 155079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6819.185225] LustreError: 155060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247467 with bad export cookie 7086422915484079836 [ 6819.190188] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.200167] LustreError: 155060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6822.882827] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6822.897328] Lustre: Skipped 2 previous similar messages [ 6825.628880] Lustre: server umount lustre-MDT0001 complete [ 6831.098728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6841.986479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.143911] LustreError: 158579:0:(llog.c:1646:llog_backup()) MGC192.168.203.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6842.148569] Lustre: 158579:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6849.506171] LustreError: 158584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.520496] LustreError: 158584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6865.870401] LustreError: 158586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.883938] LustreError: 158586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6900.192155] LustreError: 158585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6900.201589] LustreError: 158585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 6903.263393] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6903.267848] Lustre: 158579:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6943.791957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6943.795354] Lustre: Skipped 2 previous similar messages [ 6945.829308] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6946.375593] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6952.003036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.196754] LustreError: 142895:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006ecf919f x1867772926751232/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6964.217624] LustreError: 158602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.230614] LustreError: 158602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 6966.745622] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 6969.836511] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6970.090252] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6971.146981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6973.353921] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6974.418233] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6990.305038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6990.307915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.310314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6990.310321] Lustre: Skipped 8 previous similar messages [ 6990.328770] Lustre: Skipped 2 previous similar messages [ 6991.036791] Lustre: server umount lustre-MDT0000 complete [ 6993.321331] LustreError: 155058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247641 with bad export cookie 7086422915484086318 [ 6993.322940] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6993.327220] LustreError: 155058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6993.542646] Lustre: server umount lustre-MDT0001 complete [ 7005.229375] Lustre: server umount lustre-OST0000 complete [ 7016.210606] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7020.986852] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7038.613891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.737960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7038.768830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7038.820746] Lustre: lustre-MDT0000: new disk, initializing [ 7038.873159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.878545] Lustre: Skipped 1 previous similar message [ 7038.892832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7041.215249] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7046.888680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.943796] Lustre: 163425:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7047.069765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7048.917930] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7052.899322] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7054.767642] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7058.022065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7058.159505] Lustre: 165167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7061.048096] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7063.565028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7063.612373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7065.295412] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7067.219117] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7067.326656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7069.328633] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7069.433572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7073.761612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7073.768074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.770348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7073.780309] Lustre: Skipped 1 previous similar message [ 7073.787265] Lustre: Skipped 3 previous similar messages [ 7076.536328] Lustre: server umount lustre-OST0000 complete [ 7084.862329] Lustre: server umount lustre-MDT0000 complete [ 7086.764506] LustreError: 163417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247734 with bad export cookie 7086422915484103125 [ 7086.767700] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7086.771322] LustreError: 163417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7086.948691] Lustre: server umount lustre-MDT0001 complete [ 7091.276576] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:02:17 (1781247737) [ 7094.786532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.060710] LustreError: 167426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.220619] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7101.998573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.741675] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7107.989061] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7109.949617] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7113.586361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7116.772027] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7119.669390] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7134.306306] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:03:00 (1781247780) [ 7138.275158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.279604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.282121] LustreError: Skipped 1 previous similar message [ 7138.289037] Lustre: Skipped 3 previous similar messages [ 7141.993290] Lustre: server umount lustre-OST0000 complete [ 7149.794537] Lustre: server umount lustre-MDT0000 complete [ 7151.663298] LustreError: 170083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247799 with bad export cookie 7086422915484104007 [ 7151.668132] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7151.672327] LustreError: 170083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7151.826423] Lustre: server umount lustre-MDT0001 complete [ 7156.919736] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 7158.326223] Key type lgssc unregistered [ 7158.484680] LNet: 170816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7158.493500] LNetError: 170816:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7158.501794] LNet: Removed LNI 192.168.203.120@tcp [ 7158.908213] Key type .llcrypt unregistered [ 7158.909780] Key type ._llcrypt unregistered [ 7168.827531] Key type ._llcrypt registered [ 7168.829302] Key type .llcrypt registered [ 7168.880982] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7169.489108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7169.502841] alg: No test for adler32 (adler32-zlib) [ 7170.433543] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 7170.550462] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 7172.159175] Key type lgssc registered [ 7172.755520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7176.918234] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7176.928629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7192.416320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7197.536159] LustreError: 172018:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.120@tcp: failed processing log, type 4: rc = -110 [ 7202.656177] LustreError: 172018:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7202.661672] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7202.663445] LustreError: MGC192.168.203.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7202.667233] LustreError: 172018:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7202.670467] LustreError: 172018:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7202.674989] LustreError: 172018:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7202.704396] Lustre: server umount lustre-OST0000 complete [ 7202.707848] LustreError: 172018:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7206.907987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.172473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7209.897416] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7213.767983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.949494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.528766] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7218.056830] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7219.846942] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7222.881378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.988500] LustreError: 174102:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7222.994725] Lustre: 174102:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7223.062502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.593753] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7228.292504] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7229.289125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.290113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7229.294447] Lustre: Skipped 2 previous similar messages [ 7229.298046] Lustre: Skipped 1 previous similar message [ 7234.528409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.533307] Lustre: Skipped 3 previous similar messages [ 7235.435852] Lustre: server umount lustre-MDT0000 complete [ 7237.107347] LustreError: 172498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247884 with bad export cookie 3462598183041779011 [ 7237.110633] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7237.111720] LustreError: 172498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7237.267383] Lustre: server umount lustre-MDT0001 complete [ 7248.259858] Lustre: server umount lustre-OST0000 complete [ 7253.475747] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7263.746645] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7314.298163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.005464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7315.074558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7315.390309] Lustre: lustre-MDT0000: new disk, initializing [ 7315.680144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.755133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7324.797784] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7339.096339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.314083] Lustre: 177481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7339.394110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7339.406581] Lustre: Skipped 1 previous similar message [ 7339.553904] Lustre: lustre-MDT0001: new disk, initializing [ 7339.759270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7339.872609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7339.910105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7344.525295] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7351.468617] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7354.821838] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7360.915991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.196830] Lustre: lustre-OST0000: new disk, initializing [ 7361.201432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7361.216451] Lustre: 179225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7361.344039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.050949] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7370.297761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7370.303991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7370.413386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7374.187358] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7378.563569] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7378.813553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7383.039246] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7383.247933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7385.568348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7385.579256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.589101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7386.085640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7391.210300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7391.225519] Lustre: Skipped 3 previous similar messages [ 7396.326716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7396.336295] Lustre: Skipped 1 previous similar message [ 7399.391128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7399.543756] Lustre: server umount lustre-OST0000 complete [ 7406.563328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.567311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7406.575772] Lustre: Skipped 2 previous similar messages [ 7406.581906] Lustre: Skipped 1 previous similar message [ 7417.311381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7417.486810] Lustre: server umount lustre-MDT0000 complete [ 7421.102964] LustreError: 177475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248068 with bad export cookie 3462598183041779711 [ 7421.111634] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.120759] LustreError: 177475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7421.433893] Lustre: server umount lustre-MDT0001 complete [ 7431.223425] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:07:56 (1781248076) [ 7449.024693] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 7451.457315] Key type lgssc unregistered [ 7451.722937] LNet: 182008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7451.729408] LNetError: 182008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7452.775567] LNet: Removed LNI 192.168.203.120@tcp [ 7453.481177] Key type .llcrypt unregistered [ 7453.485031] Key type ._llcrypt unregistered [ 7469.563792] Key type ._llcrypt registered [ 7469.571989] Key type .llcrypt registered [ 7469.705254] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7470.509601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7470.548360] alg: No test for adler32 (adler32-zlib) [ 7471.571297] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 7471.794199] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 7473.535159] Key type lgssc registered [ 7474.659684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7481.633191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7481.660945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.117163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7486.946113] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7492.583950] LustreError: 183211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.598961] LustreError: 183211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7494.566148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7494.896145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7499.815504] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7505.453887] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7509.350574] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7517.257876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7517.473055] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7517.476850] LustreError: 184848:0:(llog.c:1687:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7517.485795] Lustre: 184848:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7517.557615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.254327] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7529.191496] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7533.025920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.033769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7533.040767] Lustre: Skipped 2 previous similar messages [ 7533.047198] Lustre: Skipped 2 previous similar messages [ 7537.475399] Lustre: server umount lustre-MDT0000 complete [ 7538.148577] LustreError: 183211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.170913] LustreError: 183211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7540.910744] LustreError: 183191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248188 with bad export cookie 12176844444082797658 [ 7540.915867] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7540.922166] LustreError: 183191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7541.178324] Lustre: server umount lustre-MDT0001 complete [ 7555.255418] Lustre: server umount lustre-OST0000 complete [ 7563.772248] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7574.002769] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7604.075082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.329973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7604.373536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7604.454333] Lustre: lustre-MDT0000: new disk, initializing [ 7604.528459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7604.545237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7608.132135] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7621.432662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.711549] Lustre: 188222:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7621.794879] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7621.796780] Lustre: Skipped 1 previous similar message [ 7621.801241] LustreError: 188989:0:(llog.c:1687:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7621.810818] LustreError: 188989:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7621.828624] Lustre: 188989:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7621.892635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7621.895481] Lustre: Skipped 1 previous similar message [ 7622.124825] Lustre: lustre-MDT0001: new disk, initializing [ 7622.302564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7622.343606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7622.356145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7628.835565] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7640.838342] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7646.937170] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7655.316839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7655.611748] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7655.614775] Lustre: Skipped 1 previous similar message [ 7655.617122] LustreError: 189944:0:(llog.c:1687:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7655.624812] LustreError: 189944:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7655.629442] Lustre: 189944:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7655.637507] Lustre: 189944:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7655.673128] Lustre: lustre-OST0000: new disk, initializing [ 7655.680501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7655.700541] Lustre: 189968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7655.858234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.114446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7661.129401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7661.257100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7662.436646] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7671.012680] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7675.323906] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7675.557982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7680.401229] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7680.573480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7687.147284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7687.150790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.152051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.182471] Lustre: Skipped 1 previous similar message [ 7688.586930] Lustre: server umount lustre-OST0000 complete [ 7693.808812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7693.817541] LustreError: Skipped 1 previous similar message [ 7693.825805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.843100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.854194] Lustre: Skipped 1 previous similar message [ 7697.379662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7697.381615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.763926] Lustre: server umount lustre-MDT0000 complete [ 7702.483193] LustreError: 188217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248350 with bad export cookie 12176844444082798379 [ 7702.489723] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7702.494917] LustreError: 188217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7702.789416] Lustre: server umount lustre-MDT0001 complete [ 7711.647232] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:12:36 (1781248356) [ 7728.883098] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7738.888611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.570476] LustreError: 192929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7739.640677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.387527] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7744.996520] LustreError: 192930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7750.114600] LustreError: 192929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7753.304820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.230552] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7761.330983] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7767.851080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7774.463991] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7783.156804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7783.323342] Lustre: lustre-OST0001: new disk, initializing [ 7783.326543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7783.331906] Lustre: 195186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7783.412407] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7783.422153] Lustre: Skipped 2 previous similar messages [ 7788.709224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7788.721415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7788.931626] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7790.685602] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7803.710350] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7811.243438] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7813.620925] Lustre: DEBUG MARKER: ADD OST3 [ 7819.633711] loop: module loaded [ 7819.676748] loop0: detected capacity change from 0 to 204800000 [ 7819.879773] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7819.897764] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7819.932313] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7826.216846] loop0: detected capacity change from 0 to 204800000 [ 7826.980250] Lustre: DEBUG MARKER: STOP OST3 [ 7832.459572] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.711453] Lustre: lustre-OST0003: new disk, initializing [ 7832.719198] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7832.726605] Lustre: 197178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7834.134183] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7834.286647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7834.293815] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7834.335899] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7836.199325] LustreError: 197333:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7836.390708] Lustre: Failing over lustre-OST0003 [ 7836.557345] Lustre: server umount lustre-OST0003 complete [ 7839.117391] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7839.224544] LustreError: 195434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.712352] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7839.718199] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.741027] Lustre: Skipped 1 previous similar message [ 7847.286445] Lustre: DEBUG MARKER: START OST3 again [ 7854.350553] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.555167] Lustre: lustre-OST0003: Not available for connect from 192.168.203.20@tcp (not set up) [ 7854.567658] Lustre: Skipped 1 previous similar message [ 7854.706938] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7854.728552] Lustre: Skipped 1 previous similar message [ 7854.758216] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7856.561071] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7856.641293] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7856.653992] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7860.674665] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 7882.220228] Lustre: Failing over lustre-OST0003 [ 7882.475227] Lustre: server umount lustre-OST0003 complete [ 7885.813660] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.841975] LustreError: 195434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.858496] LustreError: 195434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7901.158331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7901.174305] Lustre: Skipped 3 previous similar messages [ 7907.320282] Lustre: server umount lustre-MDT0000 complete [ 7912.605751] LustreError: 192910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248560 with bad export cookie 12176844444082799261 [ 7912.609907] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7912.619600] LustreError: 192910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7916.539518] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.549163] Lustre: Skipped 6 previous similar messages [ 7917.537453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7917.547804] Lustre: Skipped 7 previous similar messages [ 7919.327764] Lustre: server umount lustre-MDT0001 complete [ 7925.810727] Lustre: server umount lustre-OST0000 complete [ 7932.044986] Lustre: server umount lustre-OST0001 complete [ 7941.135385] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 7951.223865] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 7998.715038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.143520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7999.200352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7999.256577] Lustre: lustre-MDT0000: new disk, initializing [ 7999.384075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.404330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8003.900602] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8016.295324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8016.472672] Lustre: 202559:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8016.555054] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8016.559429] Lustre: Skipped 1 previous similar message [ 8016.701966] Lustre: lustre-MDT0001: new disk, initializing [ 8016.843935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8016.854986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8021.836805] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8030.275473] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8034.889314] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8043.181465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8043.605828] Lustre: 204307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8045.116353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8045.344074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8050.775927] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8059.274017] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8064.332683] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8064.626914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8069.529667] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8069.789881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8076.256348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8076.274769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.299436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8076.302138] Lustre: Skipped 1 previous similar message [ 8076.329947] Lustre: Skipped 1 previous similar message [ 8085.983544] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8086.244315] Lustre: server umount lustre-OST0000 complete [ 8096.737158] LustreError: 203423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.764180] LustreError: 203423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 8096.887863] Lustre: server umount lustre-MDT0000 complete [ 8101.236562] LustreError: 202552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248749 with bad export cookie 12176844444082801557 [ 8101.248394] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8101.263257] LustreError: 202552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8101.798173] Lustre: server umount lustre-MDT0001 complete [ 8106.526742] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:19:11 (1781248751) [ 8123.851762] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 8134.571896] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 8170.533544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.873855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8170.923125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8170.934716] Lustre: Skipped 1 previous similar message [ 8171.037876] Lustre: lustre-MDT0000: new disk, initializing [ 8171.044467] Lustre: Skipped 1 previous similar message [ 8171.212310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8171.223057] Lustre: Skipped 1 previous similar message [ 8176.374933] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8187.269426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8187.379916] Lustre: 208738:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8187.632695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8192.004845] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8199.816873] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8203.796405] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8210.523267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8210.748868] Lustre: 210483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8212.822882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8216.866787] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8224.514881] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8228.989944] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8229.193603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8233.565077] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8233.790606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8238.565226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8238.568069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.575252] LustreError: Skipped 1 previous similar message [ 8238.575341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.575344] Lustre: Skipped 4 previous similar messages [ 8238.616277] Lustre: Skipped 3 previous similar messages [ 8250.335203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8250.536040] Lustre: server umount lustre-OST0000 complete [ 8254.443307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8254.456942] LustreError: Skipped 1 previous similar message [ 8268.255176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8268.418615] Lustre: server umount lustre-MDT0000 complete [ 8269.280106] LustreError: 208751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.301944] LustreError: 208751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8271.377686] LustreError: 208731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248919 with bad export cookie 12176844444082802439 [ 8271.383163] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8271.387053] LustreError: 208731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8271.630600] Lustre: server umount lustre-MDT0001 complete [ 8278.553116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.115500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8279.120480] Lustre: Skipped 5 previous similar messages [ 8283.308704] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8291.360450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8295.236855] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8300.256680] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8303.622195] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8309.667156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.672680] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8319.446327] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8337.891581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.476667] Lustre: server umount lustre-OST0000 complete [ 8358.367231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8358.488334] Lustre: server umount lustre-MDT0000 complete [ 8361.428575] LustreError: 214914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249009 with bad export cookie 12176844444082803321 [ 8361.431678] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.442131] LustreError: 214914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8361.695105] Lustre: server umount lustre-MDT0001 complete [ 8455.025491] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 8456.989401] Key type lgssc unregistered [ 8457.234473] LNet: 215882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8457.245179] LNetError: 215882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8457.261912] LNet: Removed LNI 192.168.203.120@tcp [ 8457.919249] Key type .llcrypt unregistered [ 8457.922438] Key type ._llcrypt unregistered [ 8478.434568] Key type ._llcrypt registered [ 8478.439048] Key type .llcrypt registered [ 8478.519738] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 8491.786225] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 8492.700754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8492.729229] alg: No test for adler32 (adler32-zlib) [ 8493.769834] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 8493.959678] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 8495.593232] Key type lgssc registered [ 8496.518743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8524.155853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8524.181148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8525.412036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8525.432417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8525.500640] Lustre: lustre-MDT0000: new disk, initializing [ 8525.584854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8525.601880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8528.634294] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8536.915818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.024198] Lustre: 219200:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8537.076992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8537.081468] Lustre: Skipped 1 previous similar message [ 8537.143314] Lustre: lustre-MDT0001: new disk, initializing [ 8537.218636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8537.265071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8537.282024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8540.584615] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8547.121502] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8549.993684] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8555.230563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8555.434384] Lustre: lustre-OST0000: new disk, initializing [ 8555.438912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8555.444410] Lustre: 220941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8555.526238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8560.276337] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8564.280771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8564.300314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8564.394895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8566.024996] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8569.374716] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8569.538338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8573.149107] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8573.309723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8574.951985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.953438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8574.961288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8574.971230] Lustre: Skipped 1 previous similar message [ 8580.074050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8580.086540] Lustre: Skipped 1 previous similar message [ 8585.189136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.191532] Lustre: Skipped 1 previous similar message [ 8589.279135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8589.421446] Lustre: server umount lustre-OST0000 complete [ 8595.425357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.443697] Lustre: Skipped 1 previous similar message [ 8595.470343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.494706] Lustre: Skipped 1 previous similar message [ 8600.552318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8601.850554] Lustre: server umount lustre-MDT0000 complete [ 8607.632202] LustreError: 219191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249255 with bad export cookie 4712519111264102950 [ 8607.640844] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8607.643359] LustreError: 219191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8607.836590] Lustre: server umount lustre-MDT0001 complete [ 8635.974984] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 8648.042024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.746222] LustreError: 223736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.770757] LustreError: 223736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8648.816751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8654.309684] LustreError: 223737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8654.894546] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8658.465536] LustreError: 223736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8663.534062] LustreError: 223737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8666.563859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.977257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8672.057320] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8675.142906] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8682.722937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8683.118529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.700790] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8698.654135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8698.901745] Lustre: lustre-OST0001: new disk, initializing [ 8698.905398] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8698.912746] Lustre: 225996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8704.976828] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8709.186987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8709.194454] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8709.258658] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8715.456775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8726.230198] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8738.435191] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:29:43 (1781249383) [ 8740.329728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.333679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8740.349177] Lustre: Skipped 3 previous similar messages [ 8740.350042] Lustre: Skipped 2 previous similar messages [ 8746.527108] Lustre: server umount lustre-MDT0000 complete [ 8750.013516] LustreError: 223717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249397 with bad export cookie 4712519111264103832 [ 8750.020852] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8750.046846] LustreError: 223717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8750.265551] Lustre: server umount lustre-MDT0001 complete [ 8763.315784] Lustre: server umount lustre-OST0000 complete [ 8766.953127] Lustre: 217426:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781249398/real 1781249398] req@ffffa0327cea4000 x1867775099381504/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781249414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8767.007251] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.020689] Lustre: Skipped 1 previous similar message [ 8767.266678] Lustre: server umount lustre-OST0001 complete [ 8773.721491] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 8782.665514] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 8819.888177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.348299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8820.397960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8820.539847] Lustre: lustre-MDT0000: new disk, initializing [ 8820.762488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.774942] Lustre: Skipped 1 previous similar message [ 8820.803475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8826.015832] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8838.230519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.396601] Lustre: 229890:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8838.450087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8838.454390] Lustre: Skipped 1 previous similar message [ 8838.519310] Lustre: lustre-MDT0001: new disk, initializing [ 8838.619537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8838.627812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8843.013478] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8850.794500] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8854.388000] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8860.799432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8861.072376] Lustre: lustre-OST0000: new disk, initializing [ 8861.075142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8861.082523] Lustre: 231629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8861.173623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.180872] Lustre: Skipped 1 previous similar message [ 8862.866786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8862.880430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8862.980335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8867.697656] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8875.374841] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8880.211415] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8880.457188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8885.704671] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8885.931178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8888.805365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8888.806269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8888.812198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8888.812207] Lustre: Skipped 6 previous similar messages [ 8888.828372] Lustre: Skipped 1 previous similar message [ 8888.837603] LustreError: Skipped 1 previous similar message [ 8894.153090] Lustre: server umount lustre-OST0000 complete [ 8899.041545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.733124] Lustre: server umount lustre-MDT0000 complete [ 8904.160269] LustreError: 229901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.181488] LustreError: 229901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8906.505911] LustreError: 229881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249554 with bad export cookie 4712519111264105967 [ 8906.513152] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8906.522689] LustreError: 229881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8906.731429] Lustre: server umount lustre-MDT0001 complete [ 8913.046095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8913.374407] LustreError: 233699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8916.833362] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8923.811875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.206881] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8934.237549] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8938.588302] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8945.250394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8945.587762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.592767] Lustre: Skipped 2 previous similar messages [ 8951.640621] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 8957.700761] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9107.430248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.432169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9107.443540] Lustre: Skipped 1 previous similar message [ 9107.445331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9107.449893] LustreError: Skipped 1 previous similar message [ 9107.463495] Lustre: Skipped 6 previous similar messages [ 9109.709728] Lustre: server umount lustre-OST0000 complete [ 9113.571413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9113.583403] LustreError: Skipped 1 previous similar message [ 9119.675811] Lustre: server umount lustre-MDT0000 complete [ 9122.785182] LustreError: 233694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.799837] LustreError: 233694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9123.049445] LustreError: 233683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249770 with bad export cookie 4712519111264106849 [ 9123.057673] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9123.063471] LustreError: 233683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9123.528379] Lustre: server umount lustre-MDT0001 complete [ 9226.813776] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 9229.067251] Key type lgssc unregistered [ 9229.317533] LNet: 236948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9229.326873] LNetError: 236948:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9229.341266] LNet: Removed LNI 192.168.203.120@tcp [ 9229.957164] Key type .llcrypt unregistered [ 9229.962376] Key type ._llcrypt unregistered [ 9251.267574] Key type ._llcrypt registered [ 9251.270615] Key type .llcrypt registered [ 9251.391197] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 9264.752560] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9265.783883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9265.916744] alg: No test for adler32 (adler32-zlib) [ 9266.928791] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 9267.120529] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 9268.783167] Key type lgssc registered [ 9269.648858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9305.568314] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9313.234189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9313.254652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9314.513340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9314.540583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9314.598886] Lustre: lustre-MDT0000: new disk, initializing [ 9314.665678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9314.679788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9318.046908] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9326.871167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.016206] Lustre: 240966:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9327.076863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9327.078741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9327.082514] Lustre: Skipped 1 previous similar message [ 9327.150339] Lustre: lustre-MDT0001: new disk, initializing [ 9327.245260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9327.285275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9327.296958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9330.809588] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9335.334786] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9341.403816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.605836] Lustre: lustre-OST0000: new disk, initializing [ 9341.608876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9341.613320] Lustre: 242563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9341.706644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9346.632941] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9347.655206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9347.665350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9347.768080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9355.291921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9355.416220] Lustre: lustre-OST0001: new disk, initializing [ 9355.419643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9355.424237] Lustre: 243417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9355.508975] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9360.906650] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9364.046083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9364.057103] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9364.169845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9371.529599] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9378.294660] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9389.024674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9389.029813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9389.049593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9390.055653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9390.056681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9390.078023] Lustre: Skipped 1 previous similar message [ 9393.889681] Lustre: server umount lustre-MDT0000 complete [ 9397.158029] LustreError: 242562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250045 with bad export cookie 15742498930995169911 [ 9397.160418] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9397.173371] LustreError: 242562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9397.404926] Lustre: server umount lustre-MDT0001 complete [ 9411.200569] Lustre: server umount lustre-OST0000 complete [ 9423.421785] Lustre: server umount lustre-OST0001 complete [ 9434.166071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9441.699284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9448.079422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9455.473961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9469.058914] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9476.344104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.366242] Lustre: lustre-MDT0000: reset Object Index mappings [ 9476.807449] LustreError: 248388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.824382] LustreError: 248388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9476.865874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9479.948244] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9482.211703] LustreError: 248389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9486.316524] LustreError: 248388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9486.627356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9486.643280] Lustre: lustre-MDT0001: reset Object Index mappings [ 9486.949107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9490.437600] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9492.735249] Lustre: 249494:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9497.697553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9502.055437] LustreError: 249847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.066695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9502.355116] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9507.302016] LustreError: 249847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9508.834167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9509.031071] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9509.036853] Lustre: Skipped 1 previous similar message [ 9512.120472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9513.424947] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9519.259261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9522.030600] Lustre: 251332:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9527.783714] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9527.784660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9527.788875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9527.788884] Lustre: Skipped 3 previous similar messages [ 9527.798183] Lustre: Skipped 3 previous similar messages [ 9531.703758] Lustre: server umount lustre-MDT0000 complete [ 9532.900681] LustreError: 248388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9532.916666] LustreError: 248388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9534.555974] LustreError: 248370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250182 with bad export cookie 15742498930995176708 [ 9534.560887] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9534.563989] LustreError: 248370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9534.804737] Lustre: server umount lustre-MDT0001 complete [ 9547.358541] Lustre: server umount lustre-OST0000 complete [ 9560.866181] Lustre: server umount lustre-OST0001 complete [ 9576.950941] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:43:42 (1781250222) [ 9589.175048] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 9598.600959] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9629.103357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9629.334144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9629.372379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9629.526368] Lustre: lustre-MDT0000: new disk, initializing [ 9629.692927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9629.721298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9633.616861] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9642.638735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9642.752151] Lustre: 255556:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9642.805878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9642.809769] Lustre: Skipped 1 previous similar message [ 9642.885071] Lustre: lustre-MDT0001: new disk, initializing [ 9642.997121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9643.007218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9646.729522] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9653.483378] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9656.706720] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9662.061513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9662.307402] Lustre: lustre-OST0000: new disk, initializing [ 9662.311989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9662.320292] Lustre: 257296:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9663.583232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9663.596074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9663.684221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9666.658848] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9672.683497] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9676.675454] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9676.857480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9680.644755] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9680.819887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9684.449492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9684.457633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9684.462569] LustreError: Skipped 1 previous similar message [ 9684.463969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9684.463986] Lustre: Skipped 3 previous similar messages [ 9684.495970] Lustre: Skipped 2 previous similar messages [ 9688.610070] Lustre: server umount lustre-OST0000 complete [ 9694.176215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9694.187940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.201351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9694.204171] Lustre: Skipped 1 previous similar message [ 9697.599708] Lustre: server umount lustre-MDT0000 complete [ 9699.810146] LustreError: 255561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9699.827956] LustreError: 255561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9700.053066] LustreError: 255548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250347 with bad export cookie 15742498930995178801 [ 9700.058569] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9700.062019] LustreError: 255548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9700.287065] Lustre: server umount lustre-MDT0001 complete [ 9713.240753] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9720.907624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.321871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.330739] Lustre: Skipped 2 previous similar messages [ 9724.736896] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9731.370834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.680939] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9736.968068] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9741.983284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9747.114233] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9754.417570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9754.585873] Lustre: lustre-OST0001: new disk, initializing [ 9754.588829] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9754.592977] Lustre: 262320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9756.673468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9756.689988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9756.790484] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9760.035124] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9768.710893] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9772.217584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9777.350459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9777.402067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9787.874336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9787.884367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.903902] Lustre: Skipped 1 previous similar message [ 9787.907607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9787.912821] Lustre: Skipped 1 previous similar message [ 9789.624785] Lustre: server umount lustre-MDT0000 complete [ 9792.612113] LustreError: 260046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250440 with bad export cookie 15742498930995179683 [ 9792.617108] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9792.622041] LustreError: 260046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9792.911917] Lustre: server umount lustre-MDT0001 complete [ 9805.371835] Lustre: server umount lustre-OST0000 complete [ 9818.953585] Lustre: server umount lustre-OST0001 complete [ 9833.283386] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9841.889767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9842.315737] LustreError: 265343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.337927] LustreError: 265343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9846.576992] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9854.485384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9854.843147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9854.851699] Lustre: Skipped 4 previous similar messages [ 9858.889983] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9861.270256] Lustre: 266451:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9866.765401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.145089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9871.664373] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9878.152500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.447361] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9882.871667] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 9889.156520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9892.359266] Lustre: 268293:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9896.307574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9896.337387] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9903.899255] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:49:09 (1781250549) [ 9918.434677] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.441347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9918.451477] Lustre: Skipped 6 previous similar messages [ 9918.459578] Lustre: Skipped 7 previous similar messages [ 9923.777368] Lustre: server umount lustre-MDT0000 complete [ 9926.468977] LustreError: 265325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250574 with bad export cookie 15742498930995186445 [ 9926.472242] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9926.477396] LustreError: 265325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9928.680404] LustreError: 269001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.701789] LustreError: 269001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9932.802812] Lustre: server umount lustre-MDT0001 complete [ 9941.519860] Lustre: server umount lustre-OST0000 complete [ 9950.418929] Lustre: server umount lustre-OST0001 complete [ 9954.829565] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 9963.001629] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 9996.232509] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10003.361256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10003.548266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10003.575391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10003.631060] Lustre: lustre-MDT0000: new disk, initializing [10003.702513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10006.711184] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10015.051567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.184323] Lustre: 272479:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10015.371402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10019.033068] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10028.711727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.901815] Lustre: 274074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10030.229921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10033.180635] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10040.996603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10041.129466] Lustre: 274928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10042.609393] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10045.454918] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10053.011774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10055.704879] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10055.707665] Lustre: Skipped 1 previous similar message [10066.039091] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:51:51 (1781250711) [10068.450793] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.454529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10068.457666] Lustre: Skipped 4 previous similar messages [10068.462834] Lustre: Skipped 9 previous similar messages [10073.670531] Lustre: server umount lustre-MDT0000 complete [10075.887130] LustreError: 272470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250723 with bad export cookie 15742498930995193886 [10075.890248] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.895260] LustreError: 272470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10076.105930] Lustre: server umount lustre-MDT0001 complete [10089.047798] Lustre: server umount lustre-OST0000 complete [10101.862305] Lustre: server umount lustre-OST0001 complete [10107.896784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.254194] LustreError: 277153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.271860] LustreError: 277153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10110.728909] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10116.436142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.690058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10116.694604] Lustre: Skipped 7 previous similar messages [10119.570205] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10123.584698] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10126.294145] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10130.928362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10134.542915] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10137.954104] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10152.959184] loop1: detected capacity change from 0 to 204800000 [10153.125916] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10153.142238] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10153.153948] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10156.374107] loop1: detected capacity change from 0 to 204800000 [10159.696724] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10159.769630] LustreError: 277143:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs23950: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10159.776620] LustreError: 277143:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs23950 [10159.780251] LustreError: 277143:0:(mgs_handler.c:517:mgs_target_reg()) fs23950-MDT0001: Error checking target index: rc = -13 [10159.785191] LustreError: fs23950-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10159.789289] LustreError: 280528:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10159.792560] LustreError: 280528:0:(tgt_mount.c:2084:server_put_super()) no obd fs23950-MDT0001 [10159.795620] LustreError: 280528:0:(tgt_mount.c:116:server_deregister_mount()) fs23950-MDT0001 not registered [10159.814265] Lustre: server umount fs23950-MDT0001 complete [10159.816440] LustreError: 280528:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10164.688283] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.755549] Lustre: Setting parameter fs23950-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs23950-MDT0001 [10164.783761] Lustre: srv-fs23950-MDT0001: No data found on store. Initialize space. [10164.787372] Lustre: Skipped 4 previous similar messages [10164.835309] Lustre: fs23950-MDT0001: new disk, initializing [10164.837700] Lustre: Skipped 3 previous similar messages [10164.906990] Lustre: srv-fs23950-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10164.912075] Lustre: 280994:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23950-MDT0001-mdtlov: can't create new llog: rc = -115 [10164.920866] LustreError: 280994:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23950-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10167.429209] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10171.727820] Lustre: server umount fs23950-MDT0001 complete [10179.510519] loop1: detected capacity change from 0 to 204800000 [10182.489583] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10182.670404] Lustre: 282545:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23950-MDT0001-mdtlov: can't create new llog: rc = -115 [10182.676167] LustreError: 282545:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23950-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10184.892431] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10187.502624] Lustre: server umount fs23950-MDT0001 complete [10192.865363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.873707] Lustre: Skipped 3 previous similar messages [10197.986297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10197.990524] Lustre: Skipped 12 previous similar messages [10198.481823] Lustre: server umount lustre-MDT0000 complete [10204.197092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10204.357827] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10207.618369] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10209.778620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10212.108981] loop1: detected capacity change from 0 to 204800000 [10215.412676] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.628448] Lustre: 284955:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23950-MDT0001-mdtlov: can't create new llog: rc = -115 [10215.635870] LustreError: 284955:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23950-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10218.403327] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10221.572557] Lustre: server umount fs23950-MDT0001 complete [10223.014818] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10240.316280] Lustre: server umount lustre-OST0000 complete [10248.650299] Lustre: server umount lustre-MDT0000 complete [10251.139455] LustreError: 277133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250898 with bad export cookie 15742498930995196567 [10251.143736] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10251.145885] LustreError: 277133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10251.334526] Lustre: server umount lustre-MDT0001 complete [10257.452328] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10259.216388] Key type lgssc unregistered [10259.423965] LNet: 287261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10259.431255] LNetError: 287261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10259.450765] LNet: Removed LNI 192.168.203.120@tcp [10259.986130] Key type .llcrypt unregistered [10259.989579] Key type ._llcrypt unregistered [10272.444592] Key type ._llcrypt registered [10272.446306] Key type .llcrypt registered [10272.527863] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10273.190229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10273.230678] alg: No test for adler32 (adler32-zlib) [10274.197544] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [10274.426582] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10276.087219] Key type lgssc registered [10276.882240] Lustre: Echo OBD driver; http://www.lustre.org/ [10281.860640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10281.874955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10283.214291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10285.252286] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10290.442217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10290.691571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10293.176760] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10296.678306] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10298.852371] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10302.651479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10302.888623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10305.973349] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10309.005463] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10309.039703] LustreError: 290096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.055292] LustreError: 290096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10310.308376] LustreError: 290096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10314.208922] LustreError: 290098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.361487] LustreError: 290098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.368905] LustreError: 290098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10321.902857] loop1: detected capacity change from 0 to 204800000 [10322.030335] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10322.035297] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10322.042358] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10324.210745] loop1: detected capacity change from 0 to 204800000 [10325.476139] LustreError: 290099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.484803] LustreError: 290099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10326.802140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10326.907424] Lustre: fs3376-OST0000: new disk, initializing [10326.909809] Lustre: srv-fs3376-OST0000: No data found on store. Initialize space. [10326.915267] Lustre: 291889:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10326.958264] Lustre: fs3376-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10330.418823] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10335.715810] LustreError: 290098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.728672] LustreError: 290098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10335.990876] Lustre: server umount fs3376-OST0000 complete [10342.706214] loop1: detected capacity change from 0 to 204800000 [10346.279840] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10346.407752] Lustre: fs3376-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10350.192980] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10355.343098] loop2: detected capacity change from 0 to 204800000 [10355.452013] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10355.465604] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10355.473293] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10356.194979] LustreError: 290096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.203118] LustreError: 290096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10358.573602] loop2: detected capacity change from 0 to 204800000 [10361.973760] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10362.032351] LustreError: fs3376-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10362.046492] LustreError: fs3376-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10362.051934] LustreError: 294894:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10362.056269] LustreError: 294894:0:(tgt_mount.c:2084:server_put_super()) no obd fs3376-OST0001 [10362.062501] LustreError: 294894:0:(tgt_mount.c:116:server_deregister_mount()) fs3376-OST0001 not registered [10362.083162] Lustre: server umount fs3376-OST0001 complete [10362.085277] LustreError: 294894:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10367.917406] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10368.023694] Lustre: fs3376-OST0001: new disk, initializing [10368.028716] Lustre: srv-fs3376-OST0001: No data found on store. Initialize space. [10368.035527] Lustre: 295444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10368.097526] Lustre: fs3376-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10371.626570] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10377.274951] Lustre: server umount fs3376-OST0000 complete [10383.171181] Lustre: server umount fs3376-OST0001 complete [10391.116936] loop1: detected capacity change from 0 to 204800000 [10392.040127] LustreError: 290098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.051969] LustreError: 290098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [10392.060737] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10394.418590] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10394.539832] Lustre: fs3376-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10397.992841] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10401.952818] loop2: detected capacity change from 0 to 204800000 [10402.280854] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10405.014164] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10408.309454] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10413.104613] Lustre: server umount fs3376-OST0000 complete [10419.171301] Lustre: server umount fs3376-OST0001 complete [10426.849197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10426.857698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10426.869854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10427.872283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.875931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10427.886878] Lustre: Skipped 1 previous similar message [10432.506986] Lustre: server umount lustre-OST0000 complete [10438.113213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.114219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.124463] Lustre: Skipped 1 previous similar message [10438.133523] Lustre: Skipped 1 previous similar message [10440.959587] Lustre: server umount lustre-MDT0000 complete [10443.034080] LustreError: 288479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251090 with bad export cookie 5652670876232075766 [10443.034477] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10443.039712] LustreError: 288479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10443.233030] Lustre: server umount lustre-MDT0001 complete [10449.348793] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10450.890691] Key type lgssc unregistered [10451.096769] LNet: 301291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10451.104937] LNetError: 301291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10451.118591] LNet: Removed LNI 192.168.203.120@tcp [10451.553286] Key type .llcrypt unregistered [10451.556417] Key type ._llcrypt unregistered [10466.875902] Key type ._llcrypt registered [10466.877362] Key type .llcrypt registered [10466.951421] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [10475.633744] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10476.118995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10476.142391] alg: No test for adler32 (adler32-zlib) [10477.061578] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [10477.182205] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10478.791234] Key type lgssc registered [10479.368448] Lustre: Echo OBD driver; http://www.lustre.org/ [10498.559888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10498.572251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.715133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10499.740841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10499.798943] Lustre: lustre-MDT0000: new disk, initializing [10499.859280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10499.876049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10502.093709] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10508.622823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10508.693766] Lustre: 305090:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10508.727675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10508.731090] Lustre: Skipped 1 previous similar message [10508.779667] Lustre: lustre-MDT0001: new disk, initializing [10508.837332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10508.862242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10508.872550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10510.998649] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10515.617243] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10517.579923] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10521.367771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10521.548419] Lustre: lustre-OST0000: new disk, initializing [10521.551456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10521.556108] Lustre: 306830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10521.628427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10521.899629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10521.908349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10521.978339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10525.025820] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10529.379670] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10530.511638] Lustre: Mounted lustre-client [10530.980184] Lustre: Unmounted lustre-client [10535.907337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.910431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10535.912248] Lustre: Skipped 1 previous similar message [10535.926207] Lustre: Skipped 1 previous similar message [10537.908971] Lustre: server umount lustre-OST0000 complete [10541.026305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10541.026989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10541.036312] Lustre: Skipped 1 previous similar message [10541.046893] Lustre: Skipped 1 previous similar message [10546.144374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10546.151173] Lustre: Skipped 1 previous similar message [10551.266711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10551.273013] Lustre: Skipped 1 previous similar message [10554.335595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10554.418648] Lustre: server umount lustre-MDT0000 complete [10556.426365] LustreError: 305082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251204 with bad export cookie 7706229826810775339 [10556.428326] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10556.433995] LustreError: 305082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10556.620590] Lustre: server umount lustre-MDT0001 complete [10562.145686] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10563.884363] Key type lgssc unregistered [10583.583837] LNet: 308861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10583.589129] LNetError: 308861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10659.365793] LNet: Removed LNI 192.168.203.120@tcp [10659.921133] Key type .llcrypt unregistered [10659.922860] Key type ._llcrypt unregistered [10670.281599] Key type ._llcrypt registered [10670.284118] Key type .llcrypt registered [10670.355573] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [10679.541828] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10680.107033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10680.116249] alg: No test for adler32 (adler32-zlib) [10681.071182] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [10681.195131] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10682.815158] Key type lgssc registered [10683.435311] Lustre: Echo OBD driver; http://www.lustre.org/ [10701.109902] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:02:27 (1781251347) [10704.899084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10704.915090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10706.089100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10706.112822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10706.170657] Lustre: lustre-MDT0000: new disk, initializing [10706.216100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10706.230757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10708.877859] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10715.156714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10715.218342] Lustre: 312051:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10715.259916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10715.262971] Lustre: Skipped 1 previous similar message [10715.321236] Lustre: lustre-MDT0001: new disk, initializing [10715.377224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10715.406726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10715.418674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10717.692745] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10722.272032] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10724.246261] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10728.048884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10728.202345] Lustre: lustre-OST0000: new disk, initializing [10728.205655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10728.211894] Lustre: 313795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10728.265745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10731.371585] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10734.094664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10734.104119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10734.149238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10736.127679] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10744.289112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10744.289786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10744.297130] LustreError: Skipped 1 previous similar message [10744.298763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10744.307916] Lustre: Skipped 1 previous similar message [10744.319288] Lustre: Skipped 1 previous similar message [10747.741179] Lustre: server umount lustre-OST0000 complete [10751.456514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10751.461932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10751.468793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10754.530571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10754.532954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10754.541145] Lustre: Skipped 1 previous similar message [10759.648470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.652241] Lustre: Skipped 1 previous similar message [10764.255893] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10764.389333] Lustre: server umount lustre-MDT0000 complete [10766.464935] LustreError: 312045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251414 with bad export cookie 15177995302077579004 [10766.470305] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.473477] LustreError: 312045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10766.643571] Lustre: server umount lustre-MDT0001 complete [10772.028297] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10773.651949] Key type lgssc unregistered [10773.822492] LNet: 315422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10773.827836] LNetError: 315422:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10773.841981] LNet: Removed LNI 192.168.203.120@tcp [10774.379923] Key type .llcrypt unregistered [10774.381351] Key type ._llcrypt unregistered [10789.106454] Key type ._llcrypt registered [10789.107751] Key type .llcrypt registered [10789.162918] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10789.557383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10789.571547] alg: No test for adler32 (adler32-zlib) [10790.484355] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [10790.619375] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10792.231174] Key type lgssc registered [10792.819918] Lustre: Echo OBD driver; http://www.lustre.org/ [10796.231182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10796.241163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.435094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10799.471045] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10803.869292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.054353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10806.074264] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10808.959067] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10810.737053] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10814.412244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10814.592780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10817.603431] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10820.373650] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10834.583394] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10835.501849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10839.523562] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10839.525832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10839.535226] Lustre: Skipped 2 previous similar messages [10839.537646] Lustre: Skipped 1 previous similar message [10842.586766] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [10842.590532] Lustre: Skipped 1 previous similar message [10843.905738] Lustre: server umount lustre-MDT0000 complete [10844.642099] LustreError: 316799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10844.652878] LustreError: 316799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10845.789319] LustreError: 316785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251493 with bad export cookie 17243746005081498320 [10845.791261] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10845.796580] LustreError: 316785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10847.687196] LustreError: 316800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10847.700300] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [10849.761387] LustreError: 316798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.761808] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.788787] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10852.008913] Lustre: server umount lustre-MDT0001 complete [10860.035636] Lustre: server umount lustre-OST0000 complete [10864.761067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10867.066366] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10868.497531] Lustre: 320547:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.120@tcp [10869.567070] Lustre: server umount MGS complete [10873.724824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10873.960318] LustreError: 321118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.973769] LustreError: 321118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10873.991868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10875.700126] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10879.457110] LustreError: 321119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.108852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.298430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10882.239072] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10886.224353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10888.855300] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10891.279698] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10892.032188] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10893.813261] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10904.044465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [10909.155442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10909.156504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10909.166945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10909.175777] Lustre: Skipped 2 previous similar messages [10912.297880] Lustre: server umount lustre-OST0000 complete [10916.320414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10916.324990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10916.339744] Lustre: Skipped 1 previous similar message [10919.392719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10919.395621] Lustre: Skipped 2 previous similar messages [10920.506623] Lustre: server umount lustre-MDT0000 complete [10922.366308] LustreError: 321099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251570 with bad export cookie 17243746005081499146 [10922.367845] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10922.374381] LustreError: 321099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10922.546106] Lustre: server umount lustre-MDT0001 complete [10927.410193] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [10928.928207] Key type lgssc unregistered [10929.092656] LNet: 324288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10929.097298] LNetError: 324288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10929.108687] LNet: Removed LNI 192.168.203.120@tcp [10929.538335] Key type .llcrypt unregistered [10929.540384] Key type ._llcrypt unregistered [10938.415706] Key type ._llcrypt registered [10938.417085] Key type .llcrypt registered [10938.466652] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [10946.800047] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [10947.337436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10947.346822] alg: No test for adler32 (adler32-zlib) [10948.269488] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [10948.368504] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [10949.991432] Key type lgssc registered [10950.516854] Lustre: Echo OBD driver; http://www.lustre.org/ [10966.774047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10966.786780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10967.939611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10967.960800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10968.009437] Lustre: lustre-MDT0000: new disk, initializing [10968.060053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10968.071546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10969.962962] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10975.620625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.676271] Lustre: 327398:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10975.724394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10975.727200] Lustre: Skipped 1 previous similar message [10975.779870] Lustre: lustre-MDT0001: new disk, initializing [10975.838787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10975.864708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10975.871135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10977.888962] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [10982.498652] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10985.124886] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10990.395910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10990.635407] Lustre: lustre-OST0000: new disk, initializing [10990.638560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10990.643965] Lustre: 329138:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10990.701415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10994.768877] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11000.145436] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11000.336294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11000.341694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11000.415293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11002.955380] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11003.074824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11005.684658] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11005.816248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11010.533046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11010.534130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.536744] LustreError: Skipped 1 previous similar message [11010.537601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11010.549834] Lustre: Skipped 1 previous similar message [11015.653694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11015.656829] Lustre: Skipped 2 previous similar messages [11020.773519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11020.779037] Lustre: Skipped 1 previous similar message [11021.279172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11021.419631] Lustre: server umount lustre-OST0000 complete [11025.892078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.894212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11025.896917] Lustre: Skipped 1 previous similar message [11031.008796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11031.020203] Lustre: Skipped 2 previous similar messages [11038.175210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11038.260712] Lustre: server umount lustre-MDT0000 complete [11040.535538] LustreError: 327391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251688 with bad export cookie 14417539531941325643 [11040.539185] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11040.543059] LustreError: 327391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11040.723902] Lustre: server umount lustre-MDT0001 complete [11049.368078] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:08:14 (1781251694) [11069.464373] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [11081.066391] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [11145.927288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11146.275400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11146.323727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11146.412328] Lustre: lustre-MDT0000: new disk, initializing [11146.519428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11146.550545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11152.666995] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11165.646260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.826368] Lustre: 333724:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11165.923818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11165.927384] Lustre: Skipped 1 previous similar message [11166.047701] Lustre: lustre-MDT0001: new disk, initializing [11166.187655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11166.235438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11166.264900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11171.479160] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11179.842294] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11184.171246] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11191.354443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11191.605613] Lustre: lustre-OST0000: new disk, initializing [11191.609551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11191.617399] Lustre: 335466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11191.715974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11197.976363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11197.988834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11198.113893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11198.401652] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11205.794979] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11244.001963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11244.023701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11244.053798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11244.058623] Lustre: Skipped 2 previous similar messages [11249.638171] LustreError: 336174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11249.674929] LustreError: 336174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11249.828063] Lustre: server umount lustre-OST0000 complete [11254.754141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11254.776464] Lustre: Skipped 2 previous similar messages [11260.510668] Lustre: server umount lustre-MDT0000 complete [11264.995445] LustreError: 334789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.021709] LustreError: 334789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11265.206096] LustreError: 333716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251913 with bad export cookie 14417539531941326525 [11265.216335] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11265.223346] LustreError: 333716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11265.880710] Lustre: server umount lustre-MDT0001 complete [11279.060829] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11281.997783] Key type lgssc unregistered [11282.342526] LNet: 337513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11282.353798] LNetError: 337513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11283.429927] LNet: Removed LNI 192.168.203.120@tcp [11284.350849] Key type .llcrypt unregistered [11284.353044] Key type ._llcrypt unregistered [11385.618762] Key type ._llcrypt registered [11385.624936] Key type .llcrypt registered [11385.835248] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11386.209430] Key type .llcrypt unregistered [11386.212868] Key type ._llcrypt unregistered [11463.602228] Key type ._llcrypt registered [11463.604511] Key type .llcrypt registered [11463.684727] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11463.922198] Key type .llcrypt unregistered [11463.927501] Key type ._llcrypt unregistered [11492.809712] Key type ._llcrypt registered [11492.812151] Key type .llcrypt registered [11492.886246] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11493.092822] Key type .llcrypt unregistered [11493.094580] Key type ._llcrypt unregistered [11517.041488] Key type ._llcrypt registered [11517.043820] Key type .llcrypt registered [11517.156111] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11517.407392] Key type .llcrypt unregistered [11517.411707] Key type ._llcrypt unregistered [11573.486657] Key type ._llcrypt registered [11573.490568] Key type .llcrypt registered [11573.592204] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11573.860280] Key type .llcrypt unregistered [11573.862418] Key type ._llcrypt unregistered [11596.812790] Key type ._llcrypt registered [11596.815459] Key type .llcrypt registered [11596.960153] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11597.210801] Key type .llcrypt unregistered [11597.214801] Key type ._llcrypt unregistered [11621.224664] Key type ._llcrypt registered [11621.230703] Key type .llcrypt registered [11621.300624] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11621.515396] Key type .llcrypt unregistered [11621.519756] Key type ._llcrypt unregistered [11644.836915] Key type ._llcrypt registered [11644.838928] Key type .llcrypt registered [11644.923187] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11645.137421] Key type .llcrypt unregistered [11645.141148] Key type ._llcrypt unregistered [11667.404820] Key type ._llcrypt registered [11667.406739] Key type .llcrypt registered [11667.487080] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11667.708278] Key type .llcrypt unregistered [11667.711061] Key type ._llcrypt unregistered [11690.472184] Key type ._llcrypt registered [11690.473838] Key type .llcrypt registered [11690.559737] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11690.755475] Key type .llcrypt unregistered [11690.758198] Key type ._llcrypt unregistered [11743.758869] Key type ._llcrypt registered [11743.761082] Key type .llcrypt registered [11743.881236] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11744.131313] Key type .llcrypt unregistered [11744.135770] Key type ._llcrypt unregistered [11768.744869] Key type ._llcrypt registered [11768.746344] Key type .llcrypt registered [11768.831153] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11769.096365] Key type .llcrypt unregistered [11769.103941] Key type ._llcrypt unregistered [11792.165349] Key type ._llcrypt registered [11792.167672] Key type .llcrypt registered [11792.267131] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [11792.508495] Key type .llcrypt unregistered [11792.513380] Key type ._llcrypt unregistered [11844.312858] Key type ._llcrypt registered [11844.315780] Key type .llcrypt registered [11844.424600] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [11864.533960] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [11866.424518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11866.445812] alg: No test for adler32 (adler32-zlib) [11867.733256] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [11868.302742] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [11870.015517] Key type lgssc registered [11872.123917] Lustre: Echo OBD driver; http://www.lustre.org/ [11928.382163] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11928.429781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11929.792575] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11929.826563] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11929.887537] Lustre: tst250fs-MDT0000: new disk, initializing [11930.015480] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11930.035042] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11934.516870] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11947.720575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11947.932191] Lustre: 357370:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11948.072819] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11948.075805] Lustre: Skipped 1 previous similar message [11948.195856] Lustre: tst250fs-MDT0001: new disk, initializing [11948.380537] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11948.465600] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11948.487713] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11954.305634] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11959.401412] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11967.838068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.084961] Lustre: tst250fs-OST0000: new disk, initializing [11968.091756] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11968.100968] Lustre: 358970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11968.202036] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11974.755575] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11974.769287] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11974.975720] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11975.720350] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [11987.578861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11987.846873] Lustre: tst250fs-OST0001: new disk, initializing [11987.853157] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11987.866879] Lustre: 359826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11988.096101] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11993.716190] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11993.732769] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11993.866630] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11994.820577] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [12000.576721] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12002.631974] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12002.638769] Lustre: Skipped 1 previous similar message [12024.806897] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12024.822693] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12024.830089] Lustre: Skipped 2 previous similar messages [12028.391333] Lustre: server umount tst250fs-MDT0000 complete [12032.220278] LustreError: 357363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781252680 with bad export cookie 1326336602749533248 [12032.243285] LustreError: 357363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12032.244712] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12032.819542] Lustre: server umount tst250fs-MDT0001 complete [12046.414109] Lustre: server umount tst250fs-OST0000 complete [12060.872605] Lustre: server umount tst250fs-OST0001 complete [12080.975844] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [12090.435758] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12121.657230] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:26:07 (1781252767) [12123.281401] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12140.524150] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [12153.403972] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [12189.241102] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11904 sec ========= 04:27:14 (1781252834) [12191.118706] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:27:16 (1781252836) === [12195.402572] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:27:20 (1781252840) === [12238.505263] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [12241.580808] Key type lgssc unregistered [12241.877243] LNet: 368867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12241.888049] LNetError: 368867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12241.918362] LNet: Removed LNI 192.168.203.120@tcp [12242.867154] Key type .llcrypt unregistered [12242.872460] Key type ._llcrypt unregistered