== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:09:37 (1772474977) start mds service on oleg308-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: MODOPTS_LIBCFS= oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg308-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1772475035.88 total 10.31 last 18.91) - open/close 316 (time 1772475045.88 total 20.31 last 12.09) - open/close 521 (time 1772475056.23 total 30.66 last 19.81) - open/close 876 (time 1772475067.11 total 41.54 last 32.64) - open/close 1059 (time 1772475077.48 total 51.91 last 17.65) - open/close 1188 (time 1772475087.60 total 62.03 last 12.75) - open/close 1414 (time 1772475098.41 total 72.84 last 20.92) - open/close 1611 (time 1772475109.93 total 84.36 last 17.10) - open/close 1842 (time 1772475120.30 total 94.73 last 22.27) - open/close 2178 (time 1772475131.11 total 105.54 last 31.08) - open/close 2342 (time 1772475141.23 total 115.66 last 16.21) - open/close 2541 (time 1772475151.60 total 126.03 last 19.19) - open/close 2861 (time 1772475162.47 total 136.90 last 29.42) - open/close 3232 (time 1772475178.48 total 152.91 last 23.19) - open/close 3736 (time 1772475188.72 total 163.15 last 49.22) - open/close 3940 (time 1772475198.89 total 173.32 last 20.06) - open/close 4306 (time 1772475209.26 total 183.69 last 35.29) - open/close 4449 (time 1772475219.44 total 193.87 last 14.05) - open/close 4610 (time 1772475229.61 total 204.04 last 15.83) - open/close 4780 (time 1772475239.97 total 214.40 last 16.40) - open/close 4923 (time 1772475249.98 total 224.41 last 14.30) - open/close 5127 (time 1772475260.14 total 234.57 last 20.08) - open/close 5266 (time 1772475270.14 total 244.57 last 13.90) - open/close 5404 (time 1772475280.36 total 254.79 last 13.50) - open/close 5560 (time 1772475290.60 total 265.03 last 15.24) - open/close 5705 (time 1772475300.78 total 275.21 last 14.24) - open/close 5936 (time 1772475311.53 total 285.96 last 21.48) - open/close 6087 (time 1772475321.64 total 296.07 last 14.94) - open/close 6213 (time 1772475331.65 total 306.08 last 12.59) - open/close 6602 (time 1772475341.74 total 316.17 last 38.54) - open/close 6767 (time 1772475351.91 total 326.34 last 16.22) - open/close 6961 (time 1772475362.15 total 336.58 last 18.95) - open/close 7169 (time 1772475372.39 total 346.82 last 20.31) - open/close 7395 (time 1772475382.95 total 357.38 last 21.40) - open/close 8027 (time 1772475392.96 total 367.39 last 63.17) - open/close 10000 (time 1772475400.21 total 374.64 last 272.06) - open/close 12911 (time 1772475410.22 total 384.65 last 290.91) - open/close 15036 (time 1772475420.22 total 394.65 last 212.48) - open/close 17485 (time 1772475430.22 total 404.65 last 244.82) - open/close 19931 (time 1772475440.22 total 414.65 last 244.58) - open/close 22473 (time 1772475450.22 total 424.65 last 254.18) - open/close 24343 (time 1772475460.23 total 434.66 last 186.94) - open/close 26216 (time 1772475470.23 total 444.66 last 187.21) - open/close 28425 (time 1772475480.23 total 454.66 last 220.88) - open/close 30000 (time 1772475487.48 total 461.90 last 217.44) - open/close 32337 (time 1772475497.48 total 471.91 last 233.64) - open/close 34985 (time 1772475507.48 total 481.91 last 264.80) - open/close 37121 (time 1772475517.48 total 491.91 last 213.59) - open/close 39655 (time 1772475527.48 total 501.91 last 253.30) - open/close 42559 (time 1772475537.48 total 511.91 last 290.36) - open/close 45701 (time 1772475547.49 total 521.92 last 314.13) - open/close 48496 (time 1772475557.49 total 531.92 last 279.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 535.91 seconds: 92.61 ops/second - unlinked 0 (time 1772475562 ; total 0 ; last 0) - unlinked 10000 (time 1772475579 ; total 17 ; last 17) - unlinked 20000 (time 1772475596 ; total 34 ; last 17) - unlinked 30000 (time 1772475611 ; total 49 ; last 15) - unlinked 40000 (time 1772475627 ; total 65 ; last 16) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 81 seconds: 612.740723 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server unloading modules via unload_modules_local on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2