== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:04:51 (1741899891) start mds service on oleg415-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg415-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg415-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 896 (time 1741899930.56 total 10.70 last 83.71) - open/close 1451 (time 1741899940.73 total 20.87 last 54.58) - open/close 1933 (time 1741899950.74 total 30.88 last 48.15) - open/close 2379 (time 1741899961.69 total 41.83 last 40.75) - open/close 2936 (time 1741899971.87 total 52.01 last 54.71) - open/close 3383 (time 1741899981.87 total 62.01 last 44.69) - open/close 4264 (time 1741899992.59 total 72.73 last 82.16) - open/close 4918 (time 1741900002.87 total 83.01 last 63.65) - open/close 5640 (time 1741900013.27 total 93.41 last 69.42) - open/close 6584 (time 1741900024.17 total 104.31 last 86.60) - open/close 7369 (time 1741900034.79 total 114.93 last 73.90) - open/close 8187 (time 1741900045.46 total 125.60 last 76.68) - open/close 8957 (time 1741900056.11 total 136.25 last 72.28) - open/close 9292 (time 1741900066.83 total 146.97 last 31.25) - open/close 9877 (time 1741900082.14 total 162.28 last 38.22) - open/close 10000 (time 1741900084.41 total 164.55 last 54.29) - open/close 10827 (time 1741900095.07 total 175.21 last 77.56) - open/close 11576 (time 1741900105.56 total 185.70 last 71.37) - open/close 11976 (time 1741900116.34 total 196.48 last 37.11) - open/close 12561 (time 1741900126.51 total 206.65 last 57.52) - open/close 13051 (time 1741900136.57 total 216.71 last 48.71) - open/close 13889 (time 1741900147.22 total 227.36 last 78.70) - open/close 14856 (time 1741900157.22 total 237.36 last 96.69) - open/close 15313 (time 1741900167.22 total 247.36 last 45.70) - open/close 19903 (time 1741900177.22 total 257.36 last 459.00) - open/close 25084 (time 1741900187.22 total 267.36 last 518.05) - open/close 30000 (time 1741900196.51 total 276.65 last 529.48) - open/close 34544 (time 1741900206.51 total 286.65 last 454.28) - open/close 39383 (time 1741900216.51 total 296.65 last 483.88) - open/close 44526 (time 1741900226.51 total 306.65 last 514.25) - open/close 49405 (time 1741900236.51 total 316.65 last 487.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 317.05 seconds: 156.54 ops/second - unlinked 0 (time 1741900237 ; total 0 ; last 0) - unlinked 10000 (time 1741900247 ; total 10 ; last 10) - unlinked 20000 (time 1741900256 ; total 19 ; last 9) - unlinked 30000 (time 1741900265 ; total 28 ; last 9) - unlinked 40000 (time 1741900273 ; total 36 ; last 8) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 45 seconds: 1102.933350 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server unloading modules via unload_modules_local on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2