[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 325674114 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.521868] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.524815] pid_max: default: 32768 minimum: 301 [ 0.526520] Security Framework initialized [ 0.528121] SELinux: Initializing. [ 0.529443] SELinux: Starting in permissive mode [ 0.531265] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.536396] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.539836] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.542337] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.545320] Initializing cgroup subsys memory [ 0.546920] Initializing cgroup subsys devices [ 0.548653] Initializing cgroup subsys freezer [ 0.550390] Initializing cgroup subsys net_cls [ 0.552017] Initializing cgroup subsys blkio [ 0.553637] Initializing cgroup subsys perf_event [ 0.555296] Initializing cgroup subsys hugetlb [ 0.556937] Initializing cgroup subsys pids [ 0.558474] Initializing cgroup subsys net_prio [ 0.560480] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.563922] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.565854] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.567750] tlb_flushall_shift: 6 [ 0.569057] FEATURE SPEC_CTRL Present [ 0.570379] FEATURE IBPB_SUPPORT Present [ 0.571928] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.574249] Spectre V2 : Vulnerable [ 0.575529] Speculative Store Bypass: Vulnerable [ 0.578359] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.587447] ACPI: Core revision 20130517 [ 0.591041] ACPI: All ACPI Tables successfully acquired [ 0.593070] ftrace: allocating 30294 entries in 119 pages [ 0.651453] Enabling x2apic [ 0.652390] Enabled x2apic [ 0.653595] Switched APIC routing to physical x2apic. [ 0.657092] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.659143] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.662416] TSC deadline timer enabled [ 0.662584] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.665168] ... version: 2 [ 0.666276] ... bit width: 48 [ 0.667444] ... generic registers: 4 [ 0.668614] ... value mask: 0000ffffffffffff [ 0.670351] ... max period: 00007fffffffffff [ 0.671981] ... fixed-purpose events: 3 [ 0.673071] ... event mask: 000000070000000f [ 0.674574] KVM setup paravirtual spinlock [ 0.679748] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.683017] KVM setup async PF for cpu 1 [ 0.684328] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.686731] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.690384] KVM setup async PF for cpu 2 [ 0.677970] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.690880] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.694330] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.696029] Brought up 4 CPUs [ 0.696045] KVM setup async PF for cpu 3 [ 0.696053] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.699595] smpboot: Max logical packages: 1 [ 0.700759] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.704664] devtmpfs: initialized [ 0.705833] x86/mm: Memory block size: 128MB [ 0.710635] EVM: security.selinux [ 0.711630] EVM: security.ima [ 0.712536] EVM: security.capability [ 0.716289] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.718530] NET: Registered protocol family 16 [ 0.720126] cpuidle: using governor haltpoll [ 0.721739] ACPI: bus type PCI registered [ 0.722962] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.725147] PCI: Using configuration type 1 for base access [ 0.726848] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.735252] ACPI: Added _OSI(Module Device) [ 0.736384] ACPI: Added _OSI(Processor Device) [ 0.737555] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.738721] ACPI: Added _OSI(Processor Aggregator Device) [ 0.740187] ACPI: Added _OSI(Linux-Dell-Video) [ 0.743277] ACPI: EC: Look up EC in DSDT [ 0.746427] ACPI: Interpreter enabled [ 0.747464] ACPI: (supports S0 S3 S4 S5) [ 0.748459] ACPI: Using IOAPIC for interrupt routing [ 0.749742] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.752188] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.758160] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.759794] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.761562] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.763225] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.766705] acpiphp: Slot [2] registered [ 0.767711] acpiphp: Slot [3] registered [ 0.768795] acpiphp: Slot [4] registered [ 0.769941] acpiphp: Slot [5] registered [ 0.771013] acpiphp: Slot [6] registered [ 0.772101] acpiphp: Slot [7] registered [ 0.773186] acpiphp: Slot [8] registered [ 0.774329] acpiphp: Slot [9] registered [ 0.775456] acpiphp: Slot [10] registered [ 0.776594] acpiphp: Slot [11] registered [ 0.777718] acpiphp: Slot [12] registered [ 0.778823] acpiphp: Slot [13] registered [ 0.779925] acpiphp: Slot [14] registered [ 0.781056] acpiphp: Slot [15] registered [ 0.782161] acpiphp: Slot [16] registered [ 0.783235] acpiphp: Slot [17] registered [ 0.784311] acpiphp: Slot [18] registered [ 0.785414] acpiphp: Slot [19] registered [ 0.786544] acpiphp: Slot [20] registered [ 0.787638] acpiphp: Slot [21] registered [ 0.788729] acpiphp: Slot [22] registered [ 0.789859] acpiphp: Slot [23] registered [ 0.790944] acpiphp: Slot [24] registered [ 0.792023] acpiphp: Slot [25] registered [ 0.793109] acpiphp: Slot [26] registered [ 0.794204] acpiphp: Slot [27] registered [ 0.795224] acpiphp: Slot [28] registered [ 0.796475] acpiphp: Slot [29] registered [ 0.797547] acpiphp: Slot [30] registered [ 0.798629] acpiphp: Slot [31] registered [ 0.799654] PCI host bridge to bus 0000:00 [ 0.800674] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.802314] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.804006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.805860] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.807733] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.809605] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.811018] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.811711] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.812538] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.817555] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.819104] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.820997] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.822580] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.824358] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.826312] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.826868] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.829394] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.831684] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.833316] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.840499] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.841979] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.842905] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.848265] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.853582] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.864488] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.869110] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.872586] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.878733] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.896292] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.902190] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.905633] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.909729] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.921318] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.926310] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.931534] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.936029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.948688] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.953213] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.958004] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.961271] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.970517] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.974251] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.977292] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.985992] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.998315] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.002484] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.004086] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.007407] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.008626] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.009951] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.011658] vgaarb: loaded [ 1.012390] SCSI subsystem initialized [ 1.013252] ACPI: bus type USB registered [ 1.014158] usbcore: registered new interface driver usbfs [ 1.015928] usbcore: registered new interface driver hub [ 1.017510] usbcore: registered new device driver usb [ 1.019003] PCI: Using ACPI for IRQ routing [ 1.019826] PCI: pci_cache_line_size set to 64 bytes [ 1.020056] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.020060] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.020061] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.020371] NetLabel: Initializing [ 1.021101] NetLabel: domain hash size = 128 [ 1.021891] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.022826] NetLabel: unlabeled traffic allowed by default [ 1.024093] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.025040] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.030390] amd_nb: Cannot enumerate AMD northbridges [ 1.031509] Switched to clocksource kvm-clock [ 1.045444] pnp: PnP ACPI init [ 1.046209] ACPI: bus type PNP registered [ 1.047095] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.047163] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.047203] pnp 00:02: [dma 2] [ 1.047236] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.047324] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.047404] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.047494] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.047905] pnp: PnP ACPI: found 6 devices [ 1.049130] ACPI: bus type PNP unregistered [ 1.060435] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.060442] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.060444] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.060446] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.060448] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.060536] NET: Registered protocol family 2 [ 1.062224] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.064789] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.066914] TCP: Hash tables configured (established 32768 bind 32768) [ 1.068403] TCP: reno registered [ 1.069622] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.071528] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.073735] NET: Registered protocol family 1 [ 1.075690] RPC: Registered named UNIX socket transport module. [ 1.077404] RPC: Registered udp transport module. [ 1.078443] RPC: Registered tcp transport module. [ 1.079349] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.080693] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.082413] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.084199] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.086017] PCI: CLS 0 bytes, default 64 [ 1.086809] Unpacking initramfs... [ 2.262317] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.266122] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.268297] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.271486] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.273416] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.274846] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.276173] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.278437] cryptomgr_test (51) used greatest stack depth: 14128 bytes left [ 2.278880] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.278923] Initialise system trusted keyring [ 2.305786] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.307260] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.311486] zpool: loaded [ 2.312155] zbud: loaded [ 2.313159] VFS: Disk quotas dquot_6.6.0 [ 2.313868] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.315442] NFS: Registering the id_resolver key type [ 2.316622] Key type id_resolver registered [ 2.317556] Key type id_legacy registered [ 2.318182] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.319525] Key type big_key registered [ 2.320141] SELinux: Registering netfilter hooks [ 2.321219] cryptomgr_test (57) used greatest stack depth: 14048 bytes left [ 2.323591] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.325512] NET: Registered protocol family 38 [ 2.326517] Key type asymmetric registered [ 2.327235] Asymmetric key parser 'x509' registered [ 2.328406] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.329925] io scheduler noop registered [ 2.330612] io scheduler deadline registered (default) [ 2.332251] io scheduler cfq registered [ 2.333387] io scheduler mq-deadline registered [ 2.334120] io scheduler kyber registered [ 2.336987] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.338472] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.339629] intel_idle: does not run on family 6 model 62 [ 2.339736] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.341385] ACPI: Power Button [PWRF] [ 2.342264] GHES: HEST is not enabled! [ 2.391733] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.444924] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.569271] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.635761] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.770231] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.797899] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.827416] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.830399] Non-volatile memory driver v1.3 [ 2.831872] Linux agpgart interface v0.103 [ 2.833050] crash memory driver: version 1.1 [ 2.834750] nbd: registered device at major 43 [ 2.838312] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.838339] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.838368] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.838391] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.838413] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.844620] virtio_blk virtio1: [vda] 66840 512-byte logical blocks (34.2 MB/32.6 MiB) [ 2.848700] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 2.848772] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 2.848805] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 2.848827] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 2.848850] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 2.855404] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.860352] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 2.860386] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 2.860419] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 2.860459] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 2.860495] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 2.865826] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.872136] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 2.872185] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 2.872223] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 2.872256] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 2.872289] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 2.878285] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.886757] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 2.886826] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 2.886879] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 2.886918] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 2.886962] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 2.893466] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.899176] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 2.899314] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 2.899455] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 2.899514] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 2.899558] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 2.905960] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.912064] rdac: device handler registered [ 2.913714] hp_sw: device handler registered [ 2.914997] emc: device handler registered [ 2.916338] libphy: Fixed MDIO Bus: probed [ 2.920557] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.921617] ehci-pci: EHCI PCI platform driver [ 2.922927] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.924116] ohci-pci: OHCI PCI platform driver [ 2.925305] uhci_hcd: USB Universal Host Controller Interface driver [ 2.926703] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.930299] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.931894] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.933934] mousedev: PS/2 mouse device common for all mice [ 2.935903] rtc_cmos 00:05: RTC can wake from S4 [ 2.935974] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.941574] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 2.943924] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.947378] hidraw: raw HID events driver (C) Jiri Kosina [ 2.949264] usbcore: registered new interface driver usbhid [ 2.950937] usbhid: USB HID core driver [ 2.952001] drop_monitor: Initializing network drop monitor service [ 2.953900] Netfilter messages via NETLINK v0.30. [ 2.955392] TCP: cubic registered [ 2.956277] Initializing XFRM netlink socket [ 2.957780] NET: Registered protocol family 10 [ 2.959750] NET: Registered protocol family 17 [ 2.961122] Key type dns_resolver registered [ 2.962881] mce: Using 10 MCE banks [ 2.964150] Loading compiled-in X.509 certificates [ 2.966648] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 2.969552] registered taskstats version 1 [ 2.973230] modprobe (72) used greatest stack depth: 13456 bytes left [ 2.978075] Key type trusted registered [ 2.982837] Key type encrypted registered [ 2.983884] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 2.986818] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 2.989617] rtc_cmos 00:05: setting system clock to 2024-03-22 07:08:25 UTC (1711091305) [ 2.992756] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 2.994538] Write protecting the kernel read-only data: 12288k [ 2.995858] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 2.997079] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.005700] random: systemd: uninitialized urandom read (16 bytes read) [ 3.009168] random: systemd: uninitialized urandom read (16 bytes read) [ 3.011109] random: systemd: uninitialized urandom read (16 bytes read) [ 3.014898] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.020311] systemd[1]: Detected virtualization kvm. [ 3.021633] systemd[1]: Detected architecture x86-64. [ 3.022537] systemd[1]: Running in initial RAM disk. [ 3.025973] systemd[1]: No hostname configured. [ 3.027185] systemd[1]: Set hostname to . [ 3.028811] random: systemd: uninitialized urandom read (16 bytes read) [ 3.030896] systemd[1]: Initializing machine ID from random generator. [ 3.067482] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.069717] random: systemd: uninitialized urandom read (16 bytes read) [ 3.071144] random: systemd: uninitialized urandom read (16 bytes read) [ 3.073066] random: systemd: uninitialized urandom read (16 bytes read) [ 3.074647] random: systemd: uninitialized urandom read (16 bytes read) [ 3.077807] random: systemd: uninitialized urandom read (16 bytes read) [ 3.079874] random: systemd: uninitialized urandom read (16 bytes read) [ 3.090131] systemd[1]: Reached target Swap. [ 3.094233] systemd[1]: Created slice Root Slice. [ 3.098050] systemd[1]: Created slice System Slice. [ 3.101561] systemd[1]: Listening on udev Kernel Socket. [ 3.104997] systemd[1]: Reached target Timers. [ 3.108012] systemd[1]: Listening on udev Control Socket. [ 3.111472] systemd[1]: Reached target Slices. [ 3.114714] systemd[1]: Listening on Journal Socket. [ 3.120073] systemd[1]: Starting dracut cmdline hook... [ 3.123688] systemd[1]: Reached target Sockets. [ 3.128257] systemd[1]: Starting Journal Service... [ 3.133680] systemd[1]: Starting Setup Virtual Console... [ 3.139429] systemd[1]: Starting Load Kernel Modules... [ 3.146134] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.150083] systemd[1]: Reached target Local File Systems. [ 3.154341] systemd[1]: Started Setup Virtual Console. [ 3.160154] systemd[1]: Started Load Kernel Modules. [ 3.164333] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.170239] systemd[1]: Started Journal Service. [ 3.185946] random: fast init done [ 3.279597] tsc: Refined TSC clocksource calibration: 2399.952 MHz [ 3.615943] libata version 3.00 loaded. [ 3.617946] ata_piix 0000:00:01.1: version 2.13 [ 3.625785] scsi host0: ata_piix [ 3.632185] scsi host1: ata_piix [ 3.633675] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.635887] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 3.775172] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.828760] ip (321) used greatest stack depth: 13080 bytes left [ 3.872919] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.874476] ip (344) used greatest stack depth: 12464 bytes left [ 3.921630] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.930734] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.377996] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 7.670543] SELinux: Disabled at runtime. [ 7.671870] SELinux: Unregistering netfilter hooks [ 7.758071] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.761639] systemd[1]: Inserted module 'ip_tables' [ 8.280217] systemd-journald[566]: Received request to flush runtime journal from PID 1 [ 8.430340] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.464026] cryptd: max_cpu_qlen set to 1000 [ 8.467852] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.522478] AVX version of gcm_enc/dec engaged. [ 8.523949] AES CTR mode by8 optimization enabled [ 8.548575] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.563480] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.564579] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.565366] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.676022] EDAC MC: Ver: 3.0.0 [ 8.682435] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 8.682445] EDAC sbridge: Ver: 1.1.2 [ 10.762989] mount.nfs (772) used greatest stack depth: 10704 bytes left [ 17.790389] device-mapper: uevent: version 1.0.3 [ 17.791641] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 22.024322] libcfs: loading out-of-tree module taints kernel. [ 22.025774] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 26.956403] alg: No test for adler32 (adler32-zlib) [ 27.707628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 27.836578] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 27.991234] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 27.992697] LNet: Accept secure, port 988 [ 29.534722] Key type lgssc registered [ 29.810027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 32.583952] icp: module license 'CDDL' taints kernel. [ 32.585173] Disabling lock debugging due to kernel taint [ 35.194442] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 38.495016] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 42.985936] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 44.934442] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 44.937196] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 46.909354] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 46.912403] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 49.855948] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 52.843080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 52.870011] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 52.876000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 53.967580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 53.979112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 54.024726] Lustre: lustre-MDT0000: new disk, initializing [ 54.045291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 54.051698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 54.076782] mount.lustre (6522) used greatest stack depth: 9784 bytes left [ 54.822480] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 58.992223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.011465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 59.031823] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 59.040088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 59.042754] Lustre: Skipped 1 previous similar message [ 59.076487] Lustre: lustre-MDT0001: new disk, initializing [ 59.090836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 59.096715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 59.099813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 59.841400] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 64.415000] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 64.418431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 64.436813] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 64.440284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 64.514160] Lustre: lustre-OST0000: new disk, initializing [ 64.516128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 64.529900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 65.773810] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 69.227990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 69.230530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 69.244335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 70.110792] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 70.114335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.132881] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 70.136215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 70.164429] Lustre: lustre-OST0001: new disk, initializing [ 70.167159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 70.179543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 71.400867] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 74.432153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 74.434493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 74.442524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 76.224501] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 80.580493] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 86.213141] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing check_logdir /tmp/testlogs/ [ 86.608557] random: crng init done [ 87.033396] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing yml_node [ 87.955161] Lustre: DEBUG MARKER: Client: 2.15.61.139 [ 88.540408] Lustre: DEBUG MARKER: MDS: 2.15.61.139 [ 89.126585] Lustre: DEBUG MARKER: OSS: 2.15.61.139 [ 89.480401] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Mar 22 03:09:51 EDT 2024 [ 90.961112] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 91.305253] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 104.156047] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 104.159421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 104.164334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 108.126418] Lustre: server umount lustre-MDT0000 complete [ 109.124109] LustreError: 6532:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091411 with bad export cookie 14002027703547774972 [ 109.125127] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 109.130724] LustreError: 6532:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 109.499911] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 109.500007] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 109.500009] Lustre: Skipped 3 previous similar messages [ 109.500123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 109.500125] Lustre: Skipped 3 previous similar messages [ 109.514994] LustreError: Skipped 1 previous similar message [ 114.508013] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 114.508014] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 114.508019] Lustre: Skipped 1 previous similar message [ 114.516269] LustreError: Skipped 1 previous similar message [ 115.233351] Lustre: server umount lustre-MDT0001 complete [ 122.335257] Lustre: server umount lustre-OST0000 complete [ 129.456525] Lustre: server umount lustre-OST0001 complete [ 130.850222] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 132.479181] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 135.050453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.816403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.437277] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 138.441692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.133707] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 140.137801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.055686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.077579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.157325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 142.166286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 142.206746] Lustre: lustre-MDT0000: new disk, initializing [ 142.227827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.232673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.084796] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 146.707912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.728208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.759594] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 146.846442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 147.672881] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 150.053605] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 150.676868] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 152.460284] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 152.464337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.483013] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 152.487086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 152.581950] Lustre: lustre-OST0000: new disk, initializing [ 152.583442] Lustre: Skipped 1 previous similar message [ 152.585858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 152.589195] Lustre: Skipped 2 previous similar messages [ 152.611127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.614434] Lustre: Skipped 1 previous similar message [ 153.609059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 153.612733] Lustre: Skipped 1 previous similar message [ 153.615462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 153.635528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 154.018736] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 156.408046] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 157.276483] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 157.329122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 158.151857] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 158.198573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 163.644527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.644877] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 163.645185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 163.645186] Lustre: Skipped 1 previous similar message [ 163.654978] Lustre: Skipped 2 previous similar messages [ 164.772611] Lustre: server umount lustre-OST0000 complete [ 166.860320] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 166.864087] LustreError: Skipped 1 previous similar message [ 166.865759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.652224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.655196] Lustre: Skipped 3 previous similar messages [ 171.821413] Lustre: server umount lustre-MDT0000 complete [ 172.815299] LustreError: 16881:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091475 with bad export cookie 14002027703547776414 [ 172.816456] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 172.822144] LustreError: 16881:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 172.930568] Lustre: server umount lustre-MDT0001 complete [ 175.159034] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 03:11:17 (1711091477) [ 176.921558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.070893] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 177.087986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.883636] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 180.179428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.096381] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 182.173639] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 182.777011] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 184.585558] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 184.589981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.975306] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 187.021585] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 192.388840] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 195.292190] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 195.295318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 195.300089] Lustre: Skipped 1 previous similar message [ 195.301905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 201.214991] Lustre: server umount lustre-MDT0000 complete [ 202.240311] LustreError: 21584:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091504 with bad export cookie 14002027703547777268 [ 202.241829] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 202.248246] LustreError: 21584:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 202.352841] Lustre: server umount lustre-MDT0001 complete [ 213.404676] Lustre: 23878:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091505/real 1711091505] req@ffff88012c0e2a00 x1794209302504960/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091515 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 213.490546] Lustre: server umount lustre-OST0000 complete [ 217.590657] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 219.747783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.871740] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.884357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.886485] Lustre: Skipped 2 previous similar messages [ 220.599338] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 222.628546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.463239] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 225.561944] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 225.565682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.768563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 228.815879] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 228.818789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.839787] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 228.843241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.878643] Lustre: lustre-OST0001: new disk, initializing [ 228.880351] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 230.084193] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 230.638489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.642185] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.656548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.799419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.652288] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 242.668566] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 242.668829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 242.668830] Lustre: Skipped 3 previous similar messages [ 242.675977] Lustre: Skipped 4 previous similar messages [ 248.520027] Lustre: server umount lustre-MDT0000 complete [ 249.506458] LustreError: 28156:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091552 with bad export cookie 14002027703547778115 [ 249.508081] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 249.513275] LustreError: 28156:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 249.628318] Lustre: server umount lustre-MDT0001 complete [ 260.609717] Lustre: 30090:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091553/real 1711091553] req@ffff880085cd6680 x1794209302528192/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091563 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 260.640858] Lustre: server umount lustre-OST0000 complete [ 271.590652] Lustre: 30363:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091564/real 1711091564] req@ffff8800944b9f80 x1794209302528704/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091574 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 271.671120] Lustre: server umount lustre-OST0001 complete [ 275.043786] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 277.390057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 277.537256] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.341459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 280.526536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 281.407080] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 283.651939] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 283.655716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.030966] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 287.236885] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 287.239901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.290079] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.291994] Lustre: Skipped 6 previous similar messages [ 288.575446] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 294.836379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.713268] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 295.715073] Lustre: Skipped 1 previous similar message [ 305.644184] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.646590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.649744] Lustre: Skipped 1 previous similar message [ 305.651438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.653848] Lustre: Skipped 7 previous similar messages [ 307.925255] Lustre: server umount lustre-MDT0000 complete [ 308.896623] LustreError: 2870:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091611 with bad export cookie 14002027703547779361 [ 308.898050] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.903303] LustreError: 2870:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 309.008592] Lustre: server umount lustre-MDT0001 complete [ 320.021657] Lustre: 3700:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091612/real 1711091612] req@ffff8801391d0a80 x1794209302546944/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091622 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 320.045387] Lustre: server umount lustre-OST0000 complete [ 331.087701] Lustre: 3988:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091623/real 1711091623] req@ffff880075465f80 x1794209302547456/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091633 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 331.159295] Lustre: server umount lustre-OST0001 complete [ 334.861903] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 03:13:56 (1711091636) [ 341.004680] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 343.311145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.443996] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.228544] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 346.453205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.365035] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 349.643706] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 349.646963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 350.999390] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 353.283069] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 353.286425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.626668] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 360.895299] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 361.834809] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 371.596319] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 371.599359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.605243] Lustre: Skipped 3 previous similar messages [ 371.606943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.608828] Lustre: Skipped 4 previous similar messages [ 373.834170] Lustre: server umount lustre-MDT0000 complete [ 374.899422] LustreError: 6810:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091677 with bad export cookie 14002027703547780432 [ 374.901093] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.908482] LustreError: 6810:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.012686] Lustre: server umount lustre-MDT0001 complete [ 386.073674] Lustre: 11121:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091678/real 1711091678] req@ffff88012de2aa00 x1794209302565440/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091688 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 386.097364] Lustre: server umount lustre-OST0000 complete [ 397.178328] Lustre: server umount lustre-OST0001 complete [ 400.943632] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 03:15:02 (1711091702) [ 404.136791] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 406.577701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.697681] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.705109] LustreError: Skipped 1 previous similar message [ 407.483268] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 409.767624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.634542] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 412.856008] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 412.859295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.194620] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 416.410250] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 416.414357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.468239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.471182] Lustre: Skipped 7 previous similar messages [ 417.696898] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 423.929363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 434.892124] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 436.884851] Lustre: server umount lustre-MDT0000 complete [ 437.866417] LustreError: 14306:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091740 with bad export cookie 14002027703547781503 [ 437.867471] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.873694] LustreError: 14306:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 440.508230] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.511790] LustreError: Skipped 2 previous similar messages [ 440.511919] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.511920] Lustre: Skipped 7 previous similar messages [ 443.990960] Lustre: server umount lustre-MDT0001 complete [ 446.140451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.516946] LustreError: 3104:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012d575c00 x1794209302585920/t0(0) o250->MGC192.168.204.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 448.446468] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 450.715972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.594919] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 452.687198] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 454.098726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 454.661994] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 455.044248] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 462.019824] Lustre: server umount lustre-MDT0000 complete [ 463.095019] Lustre: server umount lustre-MDT0001 complete [ 470.106715] Lustre: server umount lustre-OST0000 complete [ 471.129045] Lustre: server umount lustre-OST0001 complete [ 474.789757] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 03:16:16 (1711091776) [ 477.825765] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 480.122471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.001635] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 483.199475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.028855] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 486.223096] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 486.228047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 487.556281] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 489.759923] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 489.763355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.061464] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 497.326534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 498.230408] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 498.232432] Lustre: Skipped 2 previous similar messages [ 507.853223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.855119] Lustre: Skipped 22 previous similar messages [ 512.881190] Lustre: server umount lustre-MDT0000 complete [ 513.876390] LustreError: 22672:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711091816 with bad export cookie 14002027703547783393 [ 513.877839] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.877841] LustreError: Skipped 1 previous similar message [ 513.887196] LustreError: 22672:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 513.974951] Lustre: server umount lustre-MDT0001 complete [ 524.932658] Lustre: 26992:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711091817/real 1711091817] req@ffff88012d576680 x1794209302617536/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711091827 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 524.940424] Lustre: 26992:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 524.958258] Lustre: server umount lustre-OST0000 complete [ 535.953200] Lustre: server umount lustre-OST0001 complete [ 539.498764] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 03:17:21 (1711091841) [ 540.112598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 542.046251] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 03:17:24 (1711091844) [ 544.398815] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 545.782492] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 547.968359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 549.481528] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 549.484651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.208770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.224267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.297678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 551.332850] Lustre: lustre-MDT0000: new disk, initializing [ 551.356476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 552.162459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 554.492528] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 556.197930] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 556.201082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 556.215516] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 556.218809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.438531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 557.448410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 557.541187] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 559.798412] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 560.268538] Lustre: DEBUG MARKER: create test files [ 567.452066] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 572.916056] Lustre: server umount lustre-OST0000 complete [ 573.992230] Lustre: server umount lustre-MDT0000 complete [ 576.832895] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 577.293533] Key type lgssc unregistered [ 577.358116] LNet: 1091:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.360790] LNet: Removed LNI 192.168.204.114@tcp [ 585.265012] alg: No test for adler32 (adler32-zlib) [ 586.016601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 586.077926] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 586.130461] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 586.132500] LNet: Accept secure, port 988 [ 587.662668] Key type lgssc registered [ 587.863367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 589.821995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 589.830025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.963153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.768359] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 592.877694] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 594.677646] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 594.681219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.794723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.807825] mount.lustre (4054) used greatest stack depth: 9544 bytes left [ 596.076959] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 597.134020] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 600.794243] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:102 to 0x240000400:129 [ 600.813652] Lustre: 4061:0:(osd_internal.h:1385:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 600.816980] Lustre: 4061:0:(osd_handler.c:1972:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 600.820282] Lustre: 4061:0:(osd_handler.c:1979:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 600.823433] Lustre: 4061:0:(osd_handler.c:1989:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 600.826786] Lustre: 4061:0:(osd_handler.c:1996:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 600.829225] Lustre: 4061:0:(osd_handler.c:2003:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 604.913650] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 606.177157] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 610.812125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.812131] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 610.820780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 613.699543] Lustre: server umount lustre-OST0000 complete [ 614.830853] Lustre: server umount lustre-MDT0000 complete [ 617.891039] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 618.406405] Key type lgssc unregistered [ 618.480996] LNet: 5711:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.484442] LNet: Removed LNI 192.168.204.114@tcp [ 627.399871] alg: No test for adler32 (adler32-zlib) [ 628.153649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.218774] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 628.268567] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 628.270648] LNet: Accept secure, port 988 [ 629.801715] Key type lgssc registered [ 630.033420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.045423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.052402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.202789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.010637] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 635.137144] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 636.933608] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 636.938121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.036638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.390627] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 639.519339] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 643.037261] Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:140 to 0x240000400:161 [ 643.053910] Lustre: 8805:0:(osd_internal.h:1385:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 643.057874] Lustre: 8805:0:(osd_handler.c:1972:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 643.061089] Lustre: 8805:0:(osd_handler.c:1979:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 643.064076] Lustre: 8805:0:(osd_handler.c:1989:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 643.066907] Lustre: 8805:0:(osd_handler.c:1996:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 643.069689] Lustre: 8805:0:(osd_handler.c:2003:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 647.315314] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 653.052246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.052257] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 653.059607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.238370] Lustre: server umount lustre-OST0000 complete [ 656.249339] Lustre: server umount lustre-MDT0000 complete [ 659.142589] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 659.654703] Key type lgssc unregistered [ 659.722928] LNet: 10385:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.726044] LNet: Removed LNI 192.168.204.114@tcp [ 668.856751] alg: No test for adler32 (adler32-zlib) [ 669.608599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.675072] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 669.726427] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 669.728146] LNet: Accept secure, port 988 [ 671.256657] Key type lgssc registered [ 671.481829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.862604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 675.734057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 677.333450] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 677.337271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 678.927886] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 678.931990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 680.747341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 680.761112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 680.767362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.839210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 681.847576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 681.881912] Lustre: lustre-MDT0000: new disk, initializing [ 681.900729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.906026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 682.691386] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 686.238583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 686.260888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.286433] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 686.300222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 686.303042] Lustre: Skipped 1 previous similar message [ 686.337027] Lustre: lustre-MDT0001: new disk, initializing [ 686.359038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 686.370526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 686.373891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 687.126712] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 689.417753] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 690.038750] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 691.829436] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 691.832528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 691.847535] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 691.851253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 691.929149] Lustre: lustre-OST0000: new disk, initializing [ 691.932221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 691.955364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.252880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 695.619911] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 696.473721] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 697.051722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 697.055150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 697.071882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 697.544230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 698.444481] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 698.505052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 702.076290] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 702.076597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.076966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 702.086792] LustreError: Skipped 1 previous similar message [ 705.103537] Lustre: server umount lustre-OST0000 complete [ 706.380229] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 706.383177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.387539] Lustre: Skipped 1 previous similar message [ 706.389631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.391799] Lustre: Skipped 1 previous similar message [ 712.092039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.094427] Lustre: Skipped 3 previous similar messages [ 712.150529] Lustre: server umount lustre-MDT0000 complete [ 713.169376] LustreError: 17681:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092015 with bad export cookie 9991737490400284752 [ 713.170384] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.178662] LustreError: 17681:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 713.294904] Lustre: server umount lustre-MDT0001 complete [ 716.846079] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 03:20:18 (1711092018) [ 718.585396] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 721.588969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 727.726103] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 729.268312] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 731.430157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 733.101084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 734.618295] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 734.621869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 736.170843] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 736.174458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.897958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.916624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.984663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 737.993389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 738.029003] Lustre: lustre-MDT0000: new disk, initializing [ 738.046575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.051784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 738.834277] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 742.253150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 742.273213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.298034] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 742.310662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 742.313559] Lustre: Skipped 1 previous similar message [ 742.346868] Lustre: lustre-MDT0001: new disk, initializing [ 742.369305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 742.379341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 742.381976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 743.127287] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 745.417258] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 745.976820] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 747.736356] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 747.739398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 747.754078] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 747.757515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.109415] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 751.381739] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 752.187686] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 755.184518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 755.187738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 755.204172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 755.283516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 756.056567] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 756.102032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 760.204143] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 760.204650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.204651] Lustre: Skipped 1 previous similar message [ 760.205005] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 760.215190] LustreError: Skipped 1 previous similar message [ 762.635141] Lustre: server umount lustre-OST0000 complete [ 765.212265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.212648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.212650] Lustre: Skipped 1 previous similar message [ 765.221259] Lustre: Skipped 2 previous similar messages [ 769.678225] Lustre: server umount lustre-MDT0000 complete [ 770.220089] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.225491] LustreError: Skipped 1 previous similar message [ 770.651413] LustreError: 25539:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092073 with bad export cookie 9991737490400285592 [ 770.652536] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.657583] LustreError: 25539:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 770.777710] Lustre: server umount lustre-MDT0001 complete [ 774.342915] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 03:21:16 (1711092076) [ 776.052050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.201955] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.220680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 776.222894] Lustre: Skipped 1 previous similar message [ 776.988175] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 779.227608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.123913] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 781.162891] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 781.748539] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 783.543259] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 783.547077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 784.913927] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 785.978645] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 786.949478] Lustre: 31331:0:(genops.c:1659:obd_export_evict_by_uuid()) MGS: evicting f9b565fb-2cb8-47cc-9efc-a1e7178c244f at adminstrative request [ 788.640300] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.644883] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 798.645363] Lustre: MGS: Client f9b565fb-2cb8-47cc-9efc-a1e7178c244f (at 0@lo) reconnecting [ 798.648480] Lustre: Evicted from MGS (at 192.168.204.114@tcp) after server handle changed from 0x8aa9c84f1c4316e0 to 0x8aa9c84f1c431956 [ 798.652175] Lustre: MGC192.168.204.114@tcp: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 818.760156] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 818.764617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 818.785037] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 818.789410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 818.834435] Lustre: lustre-OST0001: new disk, initializing [ 818.835989] Lustre: Skipped 1 previous similar message [ 818.837704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 818.839695] Lustre: Skipped 1 previous similar message [ 818.857530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 818.859493] Lustre: Skipped 2 previous similar messages [ 820.046653] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 822.354130] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 823.096477] Lustre: server umount lustre-OST0001 complete [ 825.708309] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.710973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.715717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.717962] Lustre: Skipped 1 previous similar message [ 826.015704] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.177819] Lustre: server umount lustre-OST0000 complete [ 834.396213] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 834.398386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.401964] Lustre: Skipped 1 previous similar message [ 837.218769] Lustre: server umount lustre-MDT0000 complete [ 838.181609] LustreError: 763:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092140 with bad export cookie 9991737490400287062 [ 838.182873] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.191820] LustreError: 763:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 838.309365] Lustre: server umount lustre-MDT0001 complete [ 841.828034] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 03:22:23 (1711092143) [ 842.187769] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 844.123581] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 03:22:26 (1711092146) [ 844.461629] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 846.347980] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 03:22:28 (1711092148) [ 846.719809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 848.659854] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 03:22:30 (1711092150) [ 849.037684] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 850.219251] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 850.223995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.485152] loop: module loaded [ 850.489029] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 850.495108] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 850.857842] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 851.361202] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 852.277343] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 854.474530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 855.475158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 856.441793] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 856.446439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.465722] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 857.470785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 859.150727] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 859.153822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 859.170371] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 859.173856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 859.178753] LustreError: 5943:0:(obd_config.c:797:class_setup()) setup lustre-OST0000-osd failed (-28) [ 859.181136] LustreError: 5943:0:(obd_mount.c:214:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 859.184313] LustreError: 5943:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 859.188888] LustreError: 5943:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 863.710295] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 865.270673] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 867.385903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 868.966243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 870.462380] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 870.466486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 872.003186] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 872.007857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 873.732030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 873.753342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.823977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 873.831818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 873.866361] Lustre: lustre-MDT0000: new disk, initializing [ 873.884090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.889827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 874.629150] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 878.013017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 878.032549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.128851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 878.873311] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 881.138067] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 881.703918] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 883.421526] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 883.425079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 883.442295] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 883.445430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 884.755303] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 887.030704] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 887.806877] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 891.507385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 891.523039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 891.909842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 892.689799] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 892.736194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 896.524648] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 896.524650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.524655] Lustre: Skipped 1 previous similar message [ 896.524958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 896.524959] Lustre: Skipped 5 previous similar messages [ 896.534537] LustreError: Skipped 1 previous similar message [ 899.282409] Lustre: server umount lustre-OST0000 complete [ 906.280158] Lustre: server umount lustre-MDT0000 complete [ 906.540006] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.546104] LustreError: Skipped 2 previous similar messages [ 907.239240] LustreError: 9446:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092209 with bad export cookie 9991737490400287601 [ 907.240888] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.247199] LustreError: 9446:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.343906] Lustre: server umount lustre-MDT0001 complete [ 910.828727] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 03:23:32 (1711092212) [ 912.550976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.675764] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.459577] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 915.608359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.480176] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 917.481400] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 918.009614] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 919.666861] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 919.669963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.949946] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 922.010815] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 922.825818] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 922.875148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 923.648238] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 926.737847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 928.375562] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 928.381082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 928.397293] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 928.399863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.435088] Lustre: lustre-OST0001: new disk, initializing [ 928.436736] Lustre: Skipped 2 previous similar messages [ 928.438989] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 928.441825] Lustre: Skipped 3 previous similar messages [ 929.650173] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 931.909822] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 932.710717] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 936.350962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 936.353743] Lustre: Skipped 2 previous similar messages [ 936.355264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 936.363317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 936.818393] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 937.574702] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 937.623789] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 946.382991] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 950.865733] Lustre: Failing over lustre-MDT0000 [ 950.977913] Lustre: server umount lustre-MDT0000 complete [ 951.388109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.388802] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.395521] Lustre: Skipped 5 previous similar messages [ 955.740065] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.431313] LDISKFS-fs (dm-0): recovery complete [ 964.432758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.461077] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 964.530265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 964.532291] Lustre: Skipped 6 previous similar messages [ 964.543361] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 965.269606] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 968.854190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 968.856825] Lustre: 18923:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 969.532375] Lustre: 14638:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 969.535939] Lustre: 14638:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 1 previous similar message [ 969.539628] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 969.550751] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 969.852623] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 970.257153] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 970.259798] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 970.562598] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 970.564808] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 1 previous similar message [ 970.567569] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 970.570769] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 7 previous similar messages [ 971.375315] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 971.379859] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 971.683611] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 971.685867] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 2 previous similar messages [ 972.891236] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 972.894730] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 5 previous similar messages [ 973.698600] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 973.701042] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 4 previous similar messages [ 973.703842] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 973.706380] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 5 previous similar messages [ 977.215577] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 977.220249] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 10 previous similar messages [ 977.922631] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 977.925113] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 977.929435] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 977.931959] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 10 previous similar messages [ 985.346300] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 985.349043] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 20 previous similar messages [ 986.152625] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 986.154659] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 20 previous similar messages [ 986.157292] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 986.160183] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 20 previous similar messages [ 1001.699032] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1001.702516] Lustre: 21246:0:(ldlm_lib.c:1992:extend_recovery_timer()) Skipped 42 previous similar messages [ 1002.405631] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1002.407853] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 42 previous similar messages [ 1002.411983] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1002.414405] LustreError: 21246:0:(ldlm_lib.c:2625:replay_request_or_update()) Skipped 42 previous similar messages [ 1028.774656] Lustre: 21246:0:(ldlm_lib.c:2300:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1028.776774] Lustre: 21246:0:(ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1028.778703] Lustre: 21246:0:(genops.c:1516:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0c0427e8-e5ed-461a-9eae-ce8dcb3e0674@192.168.204.14@tcp [ 1028.782057] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1028.783983] Lustre: 21246:0:(ldlm_lib.c:1824:abort_req_replay_queue()) @@@ aborted: req@ffff880087ed3100 x1794209970418752/t0(8589934752) o35->0c0427e8-e5ed-461a-9eae-ce8dcb3e0674@192.168.204.14@tcp:356/0 lens 392/0 e 0 to 0 dl 1711092341 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1028.794239] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1028.799430] Lustre: lustre-MDT0000: Denying connection for new client 0c0427e8-e5ed-461a-9eae-ce8dcb3e0674 (at 192.168.204.14@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1028.801858] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1028.808019] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1028.808069] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1028.813178] Lustre: Skipped 2 previous similar messages [ 1028.821374] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:73 to 0x280000401:577 [ 1028.824626] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:545 [ 1033.948149] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1033.950406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.953586] Lustre: Skipped 1 previous similar message [ 1033.955548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.958945] Lustre: Skipped 3 previous similar messages [ 1037.878776] Lustre: server umount lustre-OST0000 complete [ 1039.644197] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.647751] LustreError: Skipped 17 previous similar messages [ 1044.872020] Lustre: server umount lustre-OST0001 complete [ 1051.886357] Lustre: server umount lustre-MDT0000 complete [ 1052.830189] LustreError: 18041:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092355 with bad export cookie 9991737490400338715 [ 1052.831338] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1052.836747] LustreError: 18041:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1052.939517] Lustre: server umount lustre-MDT0001 complete [ 1056.393238] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 03:25:58 (1711092358) [ 1058.251607] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1058.255234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.258969] Lustre: *** cfs_fail_loc=197, val=0*** [ 1075.316930] LustreError: 24184:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 1091.319970] LustreError: 24184:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 1217.328713] Lustre: *** cfs_fail_loc=197, val=0*** [ 1217.330329] Lustre: Skipped 65 previous similar messages [ 1240.338696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.457077] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1242.468784] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1243.037534] Lustre: server umount lustre-OST0000 complete [ 1246.580631] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 03:29:08 (1711092548) [ 1259.300990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.420783] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.425407] LustreError: Skipped 2 previous similar messages [ 1260.163367] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1262.314961] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1264.141699] LustreError: 26563:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1264.146436] LustreError: 26563:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1264.150591] LustreError: 26563:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1264.250604] Lustre: server umount lustre-MDT0000 complete [ 1267.958234] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1267.962350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1271.879190] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1273.380269] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1275.469333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1276.951159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1278.368775] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1278.372201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1279.830576] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1279.833543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1281.463284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1281.479481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.545614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1281.547325] Lustre: Skipped 1 previous similar message [ 1281.553632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1281.585396] Lustre: lustre-MDT0000: new disk, initializing [ 1281.606349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1282.340446] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1285.682422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1285.699227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.784018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1286.480805] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1288.734317] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.306434] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1290.945847] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1290.948787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1290.963612] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1290.966581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.216336] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1294.439334] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1295.162130] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1295.748711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1296.219057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1296.956039] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1297.004446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1300.748374] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.748428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.748429] Lustre: Skipped 5 previous similar messages [ 1300.748825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.748827] Lustre: Skipped 7 previous similar messages [ 1300.756908] LustreError: Skipped 3 previous similar messages [ 1303.562141] Lustre: server umount lustre-OST0000 complete [ 1310.554745] Lustre: server umount lustre-MDT0000 complete [ 1311.463371] LustreError: 31505:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092613 with bad export cookie 9991737490400349082 [ 1311.464585] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.471924] LustreError: 31505:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1311.584563] Lustre: server umount lustre-MDT0001 complete [ 1315.069897] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 03:30:17 (1711092617) [ 1317.459053] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1317.941674] Key type lgssc unregistered [ 1318.010873] LNet: 4129:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.013076] LNet: Removed LNI 192.168.204.114@tcp [ 1326.081355] alg: No test for adler32 (adler32-zlib) [ 1326.831639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.887870] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1326.931208] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1326.933243] LNet: Accept secure, port 988 [ 1328.459647] Key type lgssc registered [ 1328.642162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1330.640434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1332.228484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1333.657651] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1333.660424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1335.177951] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1335.180655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1336.470395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1338.067302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1338.080435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1338.084332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.148033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1339.155456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1339.187024] Lustre: lustre-MDT0000: new disk, initializing [ 1339.203197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.208473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1339.979953] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1343.321189] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1343.325264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1343.341020] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1343.343651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1343.411080] Lustre: lustre-OST0000: new disk, initializing [ 1343.413435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1343.416419] Lustre: Skipped 1 previous similar message [ 1343.431997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.638880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1345.641085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1345.645450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1345.652571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1347.975199] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1347.978552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1347.996316] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1347.999516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.031211] Lustre: lustre-OST0001: new disk, initializing [ 1348.033338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1348.050397] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.241804] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1353.052311] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.052602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1353.057413] Lustre: Skipped 1 previous similar message [ 1358.060154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.061776] Lustre: Skipped 2 previous similar messages [ 1358.814879] Lustre: server umount lustre-MDT0000 complete [ 1360.434997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1361.554217] LustreError: 8111:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092664 with bad export cookie 258456941266598956 [ 1361.555149] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.559604] LustreError: 8111:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1361.567459] Lustre: server umount lustre-OST0000 complete [ 1362.533957] Lustre: server umount lustre-OST0001 complete [ 1366.028746] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 03:31:08 (1711092668) [ 1367.820034] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1369.668234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1369.942324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.258168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.513367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1374.719995] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 03:31:16 (1711092676) [ 1377.867514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1378.371529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1378.951430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1379.492059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1379.774819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1380.299321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1384.552751] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1386.167087] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1388.395475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.080725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1391.848691] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1391.852553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1393.522239] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1393.525096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.184011] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.200835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.261601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1395.269386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1395.302275] Lustre: lustre-MDT0000: new disk, initializing [ 1395.319926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.326288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1396.069592] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1399.499500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1399.527004] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.569252] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1399.587136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1399.589911] Lustre: Skipped 1 previous similar message [ 1399.623787] Lustre: lustre-MDT0001: new disk, initializing [ 1399.643670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1399.652192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1399.655805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1400.638989] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1403.029058] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1403.549007] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1405.167261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1405.171516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1405.188169] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1405.192083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1406.529448] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1408.310735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1408.313199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1408.332817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1408.744932] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1409.498658] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1409.543309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1410.261612] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1410.304500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1413.324353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.324456] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1413.324731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.331006] Lustre: Skipped 1 previous similar message [ 1416.832185] Lustre: server umount lustre-OST0000 complete [ 1418.332154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.336146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.337757] Lustre: Skipped 2 previous similar messages [ 1423.339950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.341512] Lustre: Skipped 1 previous similar message [ 1423.785057] Lustre: server umount lustre-MDT0000 complete [ 1424.748271] LustreError: 20226:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092727 with bad export cookie 258456941266600062 [ 1424.749580] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.754993] LustreError: 20226:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1424.850221] Lustre: server umount lustre-MDT0001 complete [ 1428.460489] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 03:32:10 (1711092730) [ 1430.237068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.365317] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.380340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.382305] Lustre: Skipped 1 previous similar message [ 1431.156479] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1433.637119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.443586] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1435.421737] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1435.989836] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1437.680896] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1437.684727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.023275] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1440.061304] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1445.612104] Lustre: *** cfs_fail_loc=159, val=0*** [ 1461.619891] Lustre: lustre-MDT0000: Client 135f10ae-ad2e-4af1-a7ad-6ae95505bbce (at 192.168.204.14@tcp) reconnecting [ 1461.904982] Lustre: *** cfs_fail_loc=159, val=0*** [ 1461.906427] Lustre: Skipped 6 previous similar messages [ 1477.911160] Lustre: lustre-MDT0000: Client 135f10ae-ad2e-4af1-a7ad-6ae95505bbce (at 192.168.204.14@tcp) reconnecting [ 1478.844185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.844905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.851705] Lustre: Skipped 2 previous similar messages [ 1478.854209] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1478.857168] LustreError: Skipped 1 previous similar message [ 1484.583327] Lustre: server umount lustre-OST0000 complete [ 1488.812248] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1488.816184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.578869] Lustre: server umount lustre-MDT0000 complete [ 1492.611288] LustreError: 24378:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092795 with bad export cookie 258456941266600902 [ 1492.612307] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.617694] LustreError: 24378:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1492.714645] Lustre: server umount lustre-MDT0001 complete [ 1495.770955] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1496.234994] Key type lgssc unregistered [ 1496.298823] LNet: 28302:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.303178] LNet: Removed LNI 192.168.204.114@tcp [ 1504.557729] alg: No test for adler32 (adler32-zlib) [ 1505.308634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.372730] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1505.418728] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1505.421448] LNet: Accept secure, port 988 [ 1506.947711] Key type lgssc registered [ 1507.146377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1508.975318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1508.980493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.123530] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.134168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.903794] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1513.726401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.857905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.858361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1515.817883] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1516.490853] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1518.300589] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1518.303489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.408316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.555832] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1520.409136] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:42 to 0x280000401:65 [ 1520.509582] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1529.790666] Lustre: *** cfs_fail_loc=159, val=0*** [ 1545.800026] Lustre: lustre-MDT0000: Client c0929a11-4bbe-40b2-ac6a-c3e78b076880 (at 192.168.204.14@tcp) reconnecting [ 1546.190284] Lustre: *** cfs_fail_loc=159, val=0*** [ 1562.194349] Lustre: lustre-MDT0001: Client c0929a11-4bbe-40b2-ac6a-c3e78b076880 (at 192.168.204.14@tcp) reconnecting [ 1562.566237] Lustre: *** cfs_fail_loc=159, val=0*** [ 1562.568457] Lustre: Skipped 4 previous similar messages [ 1578.574976] Lustre: lustre-MDT0001: Client c0929a11-4bbe-40b2-ac6a-c3e78b076880 (at 192.168.204.14@tcp) reconnecting [ 1579.283150] Lustre: 871:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1579.869939] Lustre: *** cfs_fail_loc=159, val=0*** [ 1579.871620] Lustre: Skipped 4 previous similar messages [ 1595.872200] Lustre: lustre-MDT0000: Client f331808d-d6cd-46db-bf2c-1908bfb3b307 (at 192.168.204.14@tcp) reconnecting [ 1596.120172] Lustre: *** cfs_fail_loc=159, val=0*** [ 1596.122230] Lustre: Skipped 13 previous similar messages [ 1612.125863] Lustre: lustre-MDT0000: Client f331808d-d6cd-46db-bf2c-1908bfb3b307 (at 192.168.204.14@tcp) reconnecting [ 1612.508820] Lustre: 1229:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1614.556921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.557293] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1614.557531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1614.576221] Lustre: Skipped 1 previous similar message [ 1619.564413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.567356] Lustre: Skipped 2 previous similar messages [ 1619.890495] Lustre: server umount lustre-OST0000 complete [ 1624.012959] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1624.019976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.030167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1627.420594] Lustre: server umount lustre-MDT0000 complete [ 1628.855057] LustreError: 30177:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092931 with bad export cookie 16808348563407899089 [ 1628.857789] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1628.864379] LustreError: 30177:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1629.016050] Lustre: server umount lustre-MDT0001 complete [ 1632.788485] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1633.573397] Key type lgssc unregistered [ 1633.664184] LNet: 2433:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.668877] LNet: Removed LNI 192.168.204.114@tcp [ 1644.598199] alg: No test for adler32 (adler32-zlib) [ 1645.349603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.403660] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1645.450527] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1645.452362] LNet: Accept secure, port 988 [ 1646.979793] Key type lgssc registered [ 1647.363667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1649.678409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1649.683742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.828477] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.839304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.805346] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1654.900352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.045244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1656.135281] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1657.185360] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.824693] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1659.622361] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1659.626179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1659.715466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.717754] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:85 to 0x280000401:129 [ 1661.210028] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1662.812472] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1664.720405] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 1669.583902] Lustre: 6904:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1670.107558] Lustre: 6969:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1670.513251] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1671.419872] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1671.423240] Lustre: Skipped 1 previous similar message [ 1672.027994] Lustre: 7297:0:(mdt_handler.c:84:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1674.733214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.740112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.740433] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1674.748264] Lustre: Skipped 1 previous similar message [ 1679.016305] Lustre: server umount lustre-OST0000 complete [ 1684.748560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.749119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.758935] Lustre: Skipped 2 previous similar messages [ 1686.512339] Lustre: server umount lustre-MDT0000 complete [ 1687.906820] LustreError: 7711:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711092990 with bad export cookie 9526787214250758767 [ 1687.909626] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.923453] LustreError: 7711:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1688.042767] Lustre: server umount lustre-MDT0001 complete [ 1691.838641] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1692.630541] Key type lgssc unregistered [ 1692.723153] LNet: 8507:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.728088] LNet: Removed LNI 192.168.204.114@tcp [ 1702.969243] alg: No test for adler32 (adler32-zlib) [ 1703.719604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.786231] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1703.836008] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1703.837801] LNet: Accept secure, port 988 [ 1705.366680] Key type lgssc registered [ 1705.576861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1707.991174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1707.996301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.135483] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1709.150484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.876113] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1712.098455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.187849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.003389] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1714.123502] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.849243] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1716.550604] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1716.554103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1716.641232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.996863] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1718.642068] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:85 to 0x280000401:161 [ 1719.153900] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1721.643991] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 1725.879825] Lustre: *** cfs_fail_loc=159, val=0*** [ 1741.889785] Lustre: lustre-MDT0000: Client ff143128-eb7d-4631-9904-329b5d559396 (at 192.168.204.14@tcp) reconnecting [ 1743.692235] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.695346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.700066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.685375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.685675] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.685698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.700919] Lustre: Skipped 1 previous similar message [ 1748.757570] Lustre: server umount lustre-OST0000 complete [ 1751.692331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.692591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.698141] Lustre: Skipped 1 previous similar message [ 1755.856068] Lustre: server umount lustre-MDT0000 complete [ 1756.700074] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.703239] LustreError: Skipped 1 previous similar message [ 1756.822412] LustreError: 10390:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093059 with bad export cookie 12479050407346279092 [ 1756.823978] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.827773] LustreError: 10390:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1756.934975] Lustre: server umount lustre-MDT0001 complete [ 1760.210582] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1760.688786] Key type lgssc unregistered [ 1760.752945] LNet: 14177:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.756133] LNet: Removed LNI 192.168.204.114@tcp [ 1769.220940] alg: No test for adler32 (adler32-zlib) [ 1769.974661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.033601] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1770.086530] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1770.088484] LNet: Accept secure, port 988 [ 1771.621786] Key type lgssc registered [ 1771.812476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1773.967919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1773.978420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.120462] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.131603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.059079] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1778.415624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.573089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1779.660671] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1781.121302] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1781.729687] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1783.713311] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1783.717944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.839197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.840261] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:170 to 0x280000401:193 [ 1785.231546] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1786.691360] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1788.840397] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 1804.724081] Lustre: lustre-OST0000: evicting 3e30dd0b-6725-4f85-b8e6-5682de030fcc (at 192.168.204.14@tcp) by administrative request [ 1804.728565] Lustre: Skipped 1 previous similar message [ 1828.908373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.908579] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.909007] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.914752] Lustre: Skipped 1 previous similar message [ 1833.916423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.917941] Lustre: Skipped 2 previous similar messages [ 1834.611409] Lustre: server umount lustre-OST0000 complete [ 1838.636182] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.638366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.642333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1841.652951] Lustre: server umount lustre-MDT0000 complete [ 1842.703418] LustreError: 16052:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093145 with bad export cookie 12770208352824334498 [ 1842.704602] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1842.714401] LustreError: 16052:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1842.821106] Lustre: server umount lustre-MDT0001 complete [ 1845.927218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1846.440821] Key type lgssc unregistered [ 1846.511848] LNet: 20368:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.514267] LNet: Removed LNI 192.168.204.114@tcp [ 1875.267345] alg: No test for adler32 (adler32-zlib) [ 1876.017596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1876.076499] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1876.126523] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1876.128792] LNet: Accept secure, port 988 [ 1877.654707] Key type lgssc registered [ 1877.854655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1879.692864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1879.698324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.835619] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.846258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.599018] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1883.838608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.938379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.684874] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1885.790534] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.510025] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1888.189380] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1888.193150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.287707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.513088] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1890.548339] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1891.288196] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:195 to 0x280000401:225 [ 1894.288789] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 1899.293229] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1899.293472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.294057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.871741] Lustre: server umount lustre-OST0000 complete [ 1903.948048] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1903.950725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.954756] Lustre: Skipped 1 previous similar message [ 1903.956560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1903.958681] Lustre: Skipped 1 previous similar message [ 1907.928654] Lustre: server umount lustre-MDT0000 complete [ 1908.843407] LustreError: 28852:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093211 with bad export cookie 5503862414310465120 [ 1908.844732] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1908.850830] LustreError: 28852:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1908.948500] Lustre: server umount lustre-MDT0001 complete [ 1911.641706] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1912.072715] Key type lgssc unregistered [ 1912.134805] LNet: 29647:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.137072] LNet: Removed LNI 192.168.204.114@tcp [ 1919.667588] alg: No test for adler32 (adler32-zlib) [ 1920.418592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1920.475759] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1920.521800] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1920.523207] LNet: Accept secure, port 988 [ 1922.048676] Key type lgssc registered [ 1922.225439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.786253] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1923.789165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1926.984792] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 03:40:29 (1711093229) [ 1930.551565] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1931.974913] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1934.094596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.625660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1937.023730] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1937.028006] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1938.496860] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1938.500791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1940.134402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1940.151534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1940.155326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.226793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1941.234407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1941.267731] Lustre: lustre-MDT0000: new disk, initializing [ 1941.284276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.289144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1941.980154] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1945.301793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1945.320023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.346870] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1945.359943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1945.362512] Lustre: Skipped 1 previous similar message [ 1945.399201] Lustre: lustre-MDT0001: new disk, initializing [ 1945.421671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1945.436863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1945.440012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1946.125632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1948.321566] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1948.844741] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1950.515095] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1950.518724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1950.537306] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1950.540675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1950.617451] Lustre: lustre-OST0000: new disk, initializing [ 1950.619983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1950.638217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.751210] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1953.920366] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1957.555764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1957.558086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1957.571731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1962.572152] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1962.572364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.572689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1962.579715] LustreError: Skipped 1 previous similar message [ 1967.475730] Lustre: server umount lustre-OST0000 complete [ 1970.444240] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1970.446273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.449027] Lustre: Skipped 1 previous similar message [ 1970.451034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.453056] Lustre: Skipped 1 previous similar message [ 1972.588155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.588343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.595134] Lustre: Skipped 1 previous similar message [ 1974.496373] Lustre: server umount lustre-MDT0000 complete [ 1975.426391] LustreError: 3954:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093277 with bad export cookie 996362589169131967 [ 1975.427694] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.435429] LustreError: 3954:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1975.544380] Lustre: server umount lustre-MDT0001 complete [ 1978.308650] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1978.791715] Key type lgssc unregistered [ 1978.858075] LNet: 7068:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.860384] LNet: Removed LNI 192.168.204.114@tcp [ 1987.291091] alg: No test for adler32 (adler32-zlib) [ 1988.042569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.095637] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 1988.137581] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1988.139329] LNet: Accept secure, port 988 [ 1989.663683] Key type lgssc registered [ 1989.919085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1991.877007] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1991.881571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.008564] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.017878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.958442] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1996.235693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.357427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1997.370595] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1998.451451] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1999.058692] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.747420] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2000.750466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.848390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.066265] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2002.848467] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:3 to 0x280000401:33 [ 2003.092518] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2009.732769] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2009.738235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.744538] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2009.747552] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2009.750659] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2010.407809] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2010.409472] Lustre: Skipped 2 previous similar messages [ 2010.413002] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.416348] Lustre: Skipped 2 previous similar messages [ 2010.417652] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2010.419468] Lustre: Skipped 2 previous similar messages [ 2010.420621] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2010.424974] LustreError: Skipped 2 previous similar messages [ 2010.426728] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2010.429191] Lustre: Skipped 2 previous similar messages [ 2011.678930] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2011.682754] Lustre: Skipped 4 previous similar messages [ 2011.692023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.699759] Lustre: Skipped 4 previous similar messages [ 2011.702352] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2011.705234] Lustre: Skipped 4 previous similar messages [ 2011.707039] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2011.711647] LustreError: Skipped 4 previous similar messages [ 2011.713470] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2011.716128] Lustre: Skipped 4 previous similar messages [ 2013.995327] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2013.998894] Lustre: Skipped 6 previous similar messages [ 2014.005191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.010639] Lustre: Skipped 6 previous similar messages [ 2014.012576] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2014.015860] Lustre: Skipped 6 previous similar messages [ 2014.017811] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2014.022581] LustreError: Skipped 6 previous similar messages [ 2014.025581] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2014.029221] Lustre: Skipped 6 previous similar messages [ 2018.019145] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2018.021842] Lustre: Skipped 17 previous similar messages [ 2018.026883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.029817] Lustre: Skipped 17 previous similar messages [ 2018.031031] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2018.032722] Lustre: Skipped 17 previous similar messages [ 2018.033948] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2018.037005] LustreError: Skipped 17 previous similar messages [ 2018.038480] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2018.040322] Lustre: Skipped 17 previous similar messages [ 2026.085298] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2026.086971] Lustre: Skipped 36 previous similar messages [ 2026.090441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.093148] Lustre: Skipped 36 previous similar messages [ 2026.094372] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2026.096316] Lustre: Skipped 36 previous similar messages [ 2026.097491] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2026.100255] LustreError: Skipped 36 previous similar messages [ 2026.101687] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2026.104419] Lustre: Skipped 36 previous similar messages [ 2036.630397] LustreError: 8985:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2036.633611] LustreError: 8985:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2042.416633] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2042.416957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.416959] Lustre: Skipped 59 previous similar messages [ 2042.417356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.426520] LustreError: Skipped 5 previous similar messages [ 2045.916194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.916686] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.916687] LustreError: Skipped 2 previous similar messages [ 2045.922821] Lustre: Skipped 1 previous similar message [ 2047.569590] Lustre: server umount lustre-OST0000 complete [ 2050.924549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2050.926574] Lustre: Skipped 1 previous similar message [ 2054.597702] Lustre: server umount lustre-MDT0000 complete [ 2055.643446] LustreError: 8949:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093358 with bad export cookie 17759894657738448998 [ 2055.644781] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.651692] LustreError: 8949:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2055.754155] Lustre: server umount lustre-MDT0001 complete [ 2058.644900] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2059.079189] Key type lgssc unregistered [ 2059.138878] LNet: 21774:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.141217] LNet: Removed LNI 192.168.204.114@tcp [ 2066.985754] alg: No test for adler32 (adler32-zlib) [ 2067.736607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2067.789460] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2067.833245] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2067.835325] LNet: Accept secure, port 988 [ 2069.360665] Key type lgssc registered [ 2069.542903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.317684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.322815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.442979] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.452478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.125260] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2075.245942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.345075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2076.094979] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2077.125735] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.690071] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.529191] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2079.532543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.653771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.017592] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2081.653245] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:1171 to 0x280000401:1217 [ 2082.091698] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2088.448274] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2088.923607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.928194] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2088.930409] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2088.934169] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.114@tcp (at 0@lo) [ 2088.934188] LustreError: 23687:0:(osp_precreate.c:1282:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2089.660331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.660809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.666651] Lustre: Skipped 1 previous similar message [ 2091.451998] Lustre: server umount lustre-OST0000 complete [ 2109.033629] LustreError: 23687:0:(osp_precreate.c:1282:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2118.651077] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2118.654265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.738523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.737840] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2119.742466] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2119.925482] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2120.941957] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2121.310131] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2121.884413] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012abf8000.ost_server_uuid 50 [ 2122.224554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012abf8000.ost_server_uuid in FULL state after 0 sec [ 2124.748529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.752026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2124.754500] Lustre: Skipped 1 previous similar message [ 2128.942904] Lustre: server umount lustre-OST0000 complete [ 2130.412262] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.415211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.419693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.763990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.764247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.769285] Lustre: Skipped 1 previous similar message [ 2135.944809] Lustre: server umount lustre-MDT0000 complete [ 2136.890241] LustreError: 27836:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093439 with bad export cookie 13113745336134562201 [ 2136.891130] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.895738] LustreError: 27836:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2136.994835] Lustre: server umount lustre-MDT0001 complete [ 2139.724732] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2140.178721] Key type lgssc unregistered [ 2140.242902] LNet: 29014:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.246055] LNet: Removed LNI 192.168.204.114@tcp [ 2149.398998] alg: No test for adler32 (adler32-zlib) [ 2150.149637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.211459] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2150.259180] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2150.260552] LNet: Accept secure, port 988 [ 2151.788742] Key type lgssc registered [ 2152.004628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2154.124736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2154.129648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.253670] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.263631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.029171] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2158.118133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.210941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.921670] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2161.127979] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2161.132592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.244082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.529205] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2164.782895] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2164.786392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2164.804113] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2164.807085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2164.839881] Lustre: lustre-OST0001: new disk, initializing [ 2164.842065] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2164.861699] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.863227] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:1219 to 0x280000401:1249 [ 2166.205729] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2170.027748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2170.032012] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2170.054376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2171.540491] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2177.002706] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2188.252138] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.255022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.260471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.076419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.076968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.084051] Lustre: Skipped 2 previous similar messages [ 2193.808357] Lustre: server umount lustre-MDT0000 complete [ 2194.706249] LustreError: 31302:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093497 with bad export cookie 16342499631108639545 [ 2194.707186] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.711544] LustreError: 31302:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2194.801933] Lustre: server umount lustre-MDT0001 complete [ 2205.730681] Lustre: 3492:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093498/real 1711093498] req@ffff88007a082300 x1794211527561088/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093508 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2205.756599] Lustre: server umount lustre-OST0000 complete [ 2216.884627] Lustre: 3766:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093509/real 1711093509] req@ffff88007a082300 x1794211527561472/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093519 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2216.965538] Lustre: server umount lustre-OST0001 complete [ 2217.808745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2217.834213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2218.121167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2218.145141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2218.440400] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2218.444009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2218.461363] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2218.465612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2218.753328] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2218.757057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2218.780181] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2218.784258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2221.768182] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2223.969062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.108913] LustreError: 137-5: mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.130745] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.852134] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2226.890025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.781406] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2230.081995] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2230.086308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.398817] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2233.625969] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2233.629560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.677138] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.679028] Lustre: Skipped 2 previous similar messages [ 2234.677344] Lustre: mylustre-OST0000: deleting orphan objects from 0x280000401:1251 to 0x280000401:1281 [ 2234.680975] Lustre: mylustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:33 [ 2234.909458] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2241.155826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2242.081646] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2251.708637] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.709066] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.709068] Lustre: Skipped 3 previous similar messages [ 2251.720950] Lustre: Skipped 3 previous similar messages [ 2256.716957] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2256.719548] Lustre: Skipped 6 previous similar messages [ 2256.899214] Lustre: server umount mylustre-MDT0000 complete [ 2257.911803] LustreError: 5441:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093560 with bad export cookie 16342499631108642247 [ 2257.913640] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.921301] LustreError: 5441:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2258.037203] Lustre: server umount mylustre-MDT0001 complete [ 2269.133651] Lustre: 9957:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093561/real 1711093561] req@ffff88013120dc00 x1794211527584640/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093571 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2269.155671] Lustre: server umount mylustre-OST0000 complete [ 2280.150679] Lustre: 10230:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093572/real 1711093572] req@ffff88013120dc00 x1794211527585152/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093582 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2280.232681] Lustre: server umount mylustre-OST0001 complete [ 2281.073946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2281.094638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.377038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2281.391904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.667677] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2281.671322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2281.692834] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2281.696747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.977837] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2281.981683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2282.003289] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2282.007590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2285.012239] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2287.421475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.562538] LustreError: 137-5: tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.577590] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.336903] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2290.486167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.344054] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2293.883571] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2293.889292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2295.981111] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2298.024076] LustreError: 137-5: tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.042231] Lustre: tfs-OST0000: deleting orphan objects from 0x280000401:1283 to 0x280000401:1313 [ 2301.406818] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2301.424220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2302.556509] hrtimer: interrupt took 4265044 ns [ 2303.247151] Lustre: tfs-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:65 [ 2304.646042] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2309.768256] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2312.479063] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2325.646847] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2325.649999] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.655449] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.738280] Lustre: server umount tfs-MDT0000 complete [ 2331.316220] LustreError: 12788:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093633 with bad export cookie 16342499631108644102 [ 2331.317638] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2331.334249] LustreError: 12788:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2331.633071] Lustre: server umount tfs-MDT0001 complete [ 2343.940431] Lustre: 16325:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093636/real 1711093636] req@ffff880072d29880 x1794211527605696/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093646 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2344.016908] Lustre: server umount tfs-OST0000 complete [ 2356.202650] Lustre: 16599:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711093648/real 1711093648] req@ffff880072d2a680 x1794211527606080/t0(0) o39->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1711093658 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [ 2356.322666] Lustre: server umount tfs-OST0001 complete [ 2358.046740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2358.099345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.659934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2358.723635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.217686] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2359.224564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2359.265273] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2359.273150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.729596] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2359.739279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2359.773863] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2359.781258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2365.616586] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2370.116154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.482337] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.488069] LustreError: Skipped 2 previous similar messages [ 2370.518488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.522024] Lustre: Skipped 3 previous similar messages [ 2372.140140] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2377.151296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.874995] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2383.974462] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2383.992700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.206328] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.223094] LustreError: Skipped 1 previous similar message [ 2387.249169] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2392.514632] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2392.522234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2395.476576] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2395.873022] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:1283 to 0x280000401:1345 [ 2395.873307] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:3 to 0x2c0000401:97 [ 2399.765468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2401.633317] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2412.401099] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 03:48:34 (1711093714) [ 2425.933092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.936597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.936599] Lustre: Skipped 4 previous similar messages [ 2425.945793] Lustre: Skipped 5 previous similar messages [ 2429.663548] Lustre: server umount lustre-MDT0000 complete [ 2430.806720] LustreError: 18263:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093733 with bad export cookie 16342499631108645733 [ 2430.808440] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.817342] LustreError: 18263:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2430.940076] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.943566] LustreError: Skipped 2 previous similar messages [ 2437.069412] Lustre: server umount lustre-MDT0001 complete [ 2445.171895] Lustre: server umount lustre-OST0000 complete [ 2453.161313] Lustre: server umount lustre-OST0001 complete [ 2456.064641] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2460.134956] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2465.226767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2468.924214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2472.282244] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2472.309326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2475.927523] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2475.938344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2479.348172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2479.397021] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2479.415302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2479.563979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2479.581926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2479.619375] Lustre: lustre-MDT0000: new disk, initializing [ 2479.670671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.673439] Lustre: Skipped 3 previous similar messages [ 2479.679948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2481.458218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2487.120211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2487.147398] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2487.156178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2487.226099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2487.229098] Lustre: Skipped 1 previous similar message [ 2487.271144] Lustre: lustre-MDT0001: new disk, initializing [ 2487.343934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2487.350638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2488.613332] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2493.200534] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2493.209277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2493.239069] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2493.243286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.343180] Lustre: lustre-OST0000: new disk, initializing [ 2493.347545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2494.485111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2494.488457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2494.526542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2495.205982] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2500.712968] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2500.719667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2500.759763] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2500.766890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.850382] Lustre: lustre-OST0001: new disk, initializing [ 2500.855177] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2502.336120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2502.338720] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2502.378081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2502.866521] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2510.880742] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 03:50:12 (1711093812) [ 2518.792393] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 03:50:20 (1711093820) [ 2532.417087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.423101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2532.423102] Lustre: Skipped 7 previous similar messages [ 2532.443889] Lustre: Skipped 4 previous similar messages [ 2536.684389] Lustre: server umount lustre-OST0000 complete [ 2537.425960] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.441307] LustreError: Skipped 3 previous similar messages [ 2542.413254] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2545.149451] Lustre: server umount lustre-MDT0000 complete [ 2547.421652] LustreError: 27119:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093849 with bad export cookie 16342499631108647252 [ 2547.423546] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.438321] LustreError: 27119:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2553.601235] Lustre: server umount lustre-MDT0001 complete [ 2559.009277] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2571.242179] Lustre: server umount lustre-OST0001 complete [ 2574.390122] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2581.869551] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2586.710139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2589.959196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2592.977754] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2592.987992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2596.567034] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2596.575009] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2600.169202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2600.211569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.330269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2600.333337] Lustre: Skipped 1 previous similar message [ 2600.357319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2600.398919] Lustre: lustre-MDT0000: new disk, initializing [ 2600.451417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2602.162481] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2608.367426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2608.404985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.625637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.627783] Lustre: Skipped 4 previous similar messages [ 2608.671666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2610.354978] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2614.293883] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2615.647331] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2619.510865] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2619.529494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2619.565797] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2619.574029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2619.722820] Lustre: lustre-OST0000: new disk, initializing [ 2619.726636] Lustre: Skipped 1 previous similar message [ 2619.729028] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2619.733337] Lustre: Skipped 2 previous similar messages [ 2621.602784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2621.606445] Lustre: Skipped 1 previous similar message [ 2621.609094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2621.700452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2622.546834] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2626.411665] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2631.647265] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.658620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.678407] Lustre: Skipped 5 previous similar messages [ 2631.681162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.681164] Lustre: Skipped 6 previous similar messages [ 2635.991779] Lustre: server umount lustre-OST0000 complete [ 2638.655559] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2638.658690] LustreError: Skipped 1 previous similar message [ 2644.287726] Lustre: server umount lustre-MDT0000 complete [ 2646.340334] LustreError: 3017:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711093948 with bad export cookie 16342499631108648680 [ 2646.351342] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.370032] LustreError: 3017:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2646.724460] Lustre: server umount lustre-MDT0001 complete [ 2651.922993] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2653.055140] Key type lgssc unregistered [ 2653.155436] LNet: 6910:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2653.160746] LNet: Removed LNI 192.168.204.114@tcp [ 2669.780734] alg: No test for adler32 (adler32-zlib) [ 2670.558738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2670.746477] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2670.852781] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2670.856551] LNet: Accept secure, port 988 [ 2672.452239] Key type lgssc registered [ 2673.025321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.336203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2677.351711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.694567] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.725816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.671633] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2685.786490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.978815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.933727] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2690.499894] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2691.750448] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2696.150635] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2696.161697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.429160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.087186] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2701.570053] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2702.163745] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2703.610115] Lustre: server umount lustre-OST0000 complete [ 2709.456031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.465392] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.495560] Lustre: Skipped 1 previous similar message [ 2710.267393] Lustre: server umount lustre-MDT0000 complete [ 2710.501235] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.512867] LustreError: Skipped 2 previous similar messages [ 2712.582101] LustreError: 8862:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094015 with bad export cookie 5669781906564607568 [ 2712.585955] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2712.603281] LustreError: 8862:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2712.788472] Lustre: server umount lustre-MDT0001 complete [ 2718.413433] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2720.418806] Key type lgssc unregistered [ 2720.576455] LNet: 12569:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2720.591898] LNet: Removed LNI 192.168.204.114@tcp [ 2730.895778] alg: No test for adler32 (adler32-zlib) [ 2731.746817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.916958] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2732.097785] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2732.106217] LNet: Accept secure, port 988 [ 2733.759788] Key type lgssc registered [ 2734.593838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.150450] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.165029] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2741.179639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.395159] LustreError: 13798:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 1: rc = -5 [ 2794.424841] LustreError: 13798:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 2944.537944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.517350] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2950.578438] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2952.095865] Lustre: server umount lustre-OST0000 complete [ 2962.986019] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2965.150034] Key type lgssc unregistered [ 2965.402157] LNet: 15628:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2965.405668] LNet: Removed LNI 192.168.204.114@tcp [ 2998.299428] alg: No test for adler32 (adler32-zlib) [ 2999.052817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2999.218774] Lustre: Lustre: Build Version: 2.15.61_139_g36c911c [ 2999.327664] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2999.331355] LNet: Accept secure, port 988 [ 3000.918603] Key type lgssc registered [ 3001.507814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3008.798922] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.603927] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.364360] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3010.376796] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3011.065039] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3011.074321] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3011.693842] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3014.067712] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3016.187393] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3016.199375] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3018.385059] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 3018.393268] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3020.708518] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3020.733083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3020.743652] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.778901] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3020.783436] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3020.795232] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3020.800390] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3020.809153] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3020.815606] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3020.826426] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3020.832210] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3020.840843] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3020.849941] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3020.856335] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3020.861530] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3020.871346] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3020.879115] Lustre: 20557:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3024.207893] Lustre: server umount MGS complete [ 3024.653087] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.876104] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.908565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.941031] Lustre: lustre-MDT0000: Aborting client recovery [ 3025.434305] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.477400] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.501348] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3025.506565] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3025.514806] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3025.523807] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3025.531366] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3025.538719] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3025.545674] Lustre: 21106:0:(osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3025.559447] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3025.658085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3025.687856] Lustre: lustre-MDT0001: Aborting client recovery [ 3026.165915] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 3026.173155] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.204986] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 3026.210376] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.226954] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3026.234315] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3026.240958] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3026.251517] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3026.259682] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3026.266636] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3026.275455] Lustre: 21200:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3026.345608] Lustre: Found index 0 for lustre-OST0000, updating log [ 3026.403231] Lustre: lustre-OST0000: Aborting recovery [ 3026.863667] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 3026.870054] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.897488] LDISKFS-fs (loop3): file extents enabled, maximum tree depth=5 [ 3026.904192] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.932707] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3026.937200] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3026.944475] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3026.950198] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3026.960436] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3026.968367] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3026.974568] Lustre: 21294:0:(osd_scrub.c:1818:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3027.021802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3027.026781] Lustre: Skipped 1 previous similar message [ 3027.038233] Lustre: lustre-OST0001: Aborting recovery [ 3032.043448] LustreError: 21037:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 3035.036393] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 3035.039796] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 3035.050500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3035.060548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3036.040353] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 3036.043469] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 3037.906317] LustreError: 21007:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 877us: rc = -2 [ 3050.669161] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3050.678100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.687435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3055.043642] Lustre: server umount lustre-MDT0000 complete [ 3055.603238] LustreError: 20992:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094358 with bad export cookie 12197495060144566976 [ 3055.603403] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.614115] LustreError: 20992:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3056.077674] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.078135] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.080378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3056.080379] Lustre: Skipped 4 previous similar messages [ 3056.097397] Lustre: Skipped 4 previous similar messages [ 3060.087223] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.087533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3060.087534] Lustre: Skipped 1 previous similar message [ 3060.121393] LustreError: Skipped 2 previous similar messages [ 3061.827804] Lustre: server umount lustre-MDT0001 complete [ 3068.324320] Lustre: server umount lustre-OST0000 complete [ 3074.953562] Lustre: server umount lustre-OST0001 complete [ 3081.539278] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 03:59:43 (1711094383) [ 3094.495332] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3098.151798] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3103.209597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3106.995665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.233482] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3110.240731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.525355] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3113.538985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.277091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.306278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.428626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3117.443410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3117.488004] Lustre: lustre-MDT0000: new disk, initializing [ 3117.537849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.552039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3119.370652] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3125.554813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3125.591402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.640350] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3125.673785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3125.676478] Lustre: Skipped 1 previous similar message [ 3125.748446] Lustre: lustre-MDT0001: new disk, initializing [ 3125.820974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3125.845181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3125.850251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3127.574559] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3131.206162] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3132.496431] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3136.115697] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3136.128468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3136.177333] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3136.187547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3136.323322] Lustre: lustre-OST0000: new disk, initializing [ 3136.326127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3136.371301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.414732] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3140.874609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3140.878866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3140.949157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3142.063696] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3145.353110] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3151.378920] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3156.557030] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3195.999350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.999863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3195.999865] Lustre: Skipped 1 previous similar message [ 3196.009698] Lustre: Skipped 1 previous similar message [ 3201.030254] Lustre: server umount lustre-OST0000 complete [ 3205.916630] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3205.919805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.925629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.928078] Lustre: Skipped 1 previous similar message [ 3209.255921] Lustre: server umount lustre-MDT0000 complete [ 3211.021292] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.038222] LustreError: Skipped 1 previous similar message [ 3211.704083] LustreError: 26778:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711094514 with bad export cookie 12197495060144569643 [ 3211.708356] LustreError: 166-1: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3211.742445] LustreError: 26778:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3211.936830] Lustre: server umount lustre-MDT0001 complete [ 3217.368472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.056262] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3221.952965] Lustre: server umount MGS complete [ 3229.149998] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.404035] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.415122] LustreError: 1221:0:(qmt_pool.c:1518:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3229.458179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.592311] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3237.345452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.775864] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3242.403907] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3243.880056] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3247.789565] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3247.802658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3250.987648] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3253.437214] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3253.701181] Lustre: MGS: Client f0b92257-2b25-40c7-9e2c-b1c0f0703802 (at 192.168.204.14@tcp) reconnecting [ 3255.432986] Lustre: DEBUG MARKER: conf-sanity test_109a: @@@@@@ FAIL: wrong llite parameter after clear_conf