== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:42:11 (1779691331) start mds service on oleg313-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg313-server: detected 4 online CPUs by sysfs oleg313-server: MODOPTS_LIBCFS= oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg313-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg313-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg313-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 298 (time 1779691404.08 total 13.03 last 22.87) - open/close 633 (time 1779691414.08 total 23.03 last 33.50) - open/close 923 (time 1779691425.78 total 34.74 last 24.78) - open/close 1227 (time 1779691435.79 total 44.74 last 30.39) - open/close 1383 (time 1779691446.46 total 55.41 last 14.61) - open/close 1660 (time 1779691459.32 total 68.27 last 21.55) - open/close 1740 (time 1779691469.32 total 78.27 last 8.00) - open/close 1967 (time 1779691479.86 total 88.81 last 21.54) - open/close 2141 (time 1779691490.81 total 99.76 last 15.88) - open/close 2459 (time 1779691501.69 total 110.64 last 29.24) - open/close 2733 (time 1779691512.19 total 121.14 last 26.10) - open/close 2910 (time 1779691522.19 total 131.14 last 17.70) - open/close 3232 (time 1779691532.19 total 141.14 last 32.20) - open/close 3579 (time 1779691542.83 total 151.79 last 32.59) - open/close 3931 (time 1779691552.84 total 161.79 last 35.20) - open/close 4145 (time 1779691562.84 total 171.80 last 21.38) - open/close 4498 (time 1779691573.75 total 182.70 last 32.38) - open/close 4845 (time 1779691584.44 total 193.39 last 32.44) - open/close 5191 (time 1779691594.45 total 203.40 last 34.59) - open/close 5469 (time 1779691604.53 total 213.48 last 27.58) - open/close 5670 (time 1779691615.09 total 224.04 last 19.03) - open/close 5944 (time 1779691625.26 total 234.22 last 26.93) - open/close 6216 (time 1779691635.31 total 244.26 last 27.07) - open/close 6468 (time 1779691646.32 total 255.27 last 22.90) - open/close 6652 (time 1779691656.88 total 265.83 last 17.42) - open/close 6889 (time 1779691667.31 total 276.26 last 22.72) - open/close 7087 (time 1779691677.49 total 286.44 last 19.45) - open/close 7281 (time 1779691687.98 total 296.93 last 18.49) - open/close 7586 (time 1779691697.98 total 306.94 last 30.50) - open/close 7723 (time 1779691708.91 total 317.86 last 12.54) - open/close 7932 (time 1779691719.61 total 328.56 last 19.54) - open/close 8218 (time 1779691729.90 total 338.86 last 27.78) - open/close 8461 (time 1779691740.93 total 349.88 last 22.05) - open/close 8752 (time 1779691750.93 total 359.89 last 29.08) - open/close 8892 (time 1779691761.01 total 369.96 last 13.89) - open/close 9160 (time 1779691771.01 total 379.96 last 26.80) - open/close 9355 (time 1779691781.49 total 390.44 last 18.61) - open/close 9580 (time 1779691792.24 total 401.20 last 20.92) - open/close 9716 (time 1779691802.36 total 411.31 last 13.45) - open/close 10000 (time 1779691812.30 total 421.25 last 28.56) - open/close 10262 (time 1779691823.28 total 432.23 last 23.86) - open/close 11390 (time 1779691833.28 total 442.24 last 112.77) - open/close 13301 (time 1779691843.29 total 452.24 last 191.05) - open/close 14943 (time 1779691853.29 total 462.24 last 164.14) - open/close 16600 (time 1779691863.29 total 472.24 last 165.68) - open/close 18227 (time 1779691873.29 total 482.25 last 162.65) - open/close 19932 (time 1779691883.30 total 492.25 last 170.45) - open/close 21625 (time 1779691893.30 total 502.25 last 169.23) - open/close 22996 (time 1779691903.31 total 512.27 last 136.94) - open/close 24457 (time 1779691913.32 total 522.27 last 146.08) - open/close 25962 (time 1779691923.32 total 532.27 last 150.46) - open/close 28004 (time 1779691933.32 total 542.27 last 204.16) - open/close 29963 (time 1779691943.32 total 552.28 last 195.85) - open/close 32063 (time 1779691953.33 total 562.28 last 209.92) - open/close 33865 (time 1779691963.33 total 572.28 last 180.16) - open/close 35324 (time 1779691973.33 total 582.28 last 145.89) - open/close 36926 (time 1779691983.33 total 592.28 last 160.20) - open/close 38606 (time 1779691993.33 total 602.28 last 167.97) - open/close 40000 (time 1779692000.19 total 609.14 last 203.24) - open/close 41890 (time 1779692010.19 total 619.14 last 188.99) - open/close 44019 (time 1779692020.19 total 629.15 last 212.85) - open/close 45864 (time 1779692030.20 total 639.15 last 184.45) - open/close 48124 (time 1779692040.20 total 649.15 last 225.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 656.30 seconds: 75.62 ops/second - unlinked 0 (time 1779692049 ; total 0 ; last 0) - unlinked 10000 (time 1779692081 ; total 32 ; last 32) - unlinked 20000 (time 1779692105 ; total 56 ; last 24) - unlinked 30000 (time 1779692125 ; total 76 ; last 20) - unlinked 40000 (time 1779692144 ; total 95 ; last 19) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 116 seconds: 427.853455 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg313-server unloading modules via unload_modules_local on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local oleg313-server: modules unloaded. pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2