== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:11:47 (1787789507) start mds service on oleg331-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 194 (time 1787789576.24 total 10.01 last 19.39) - open/close 383 (time 1787789586.26 total 20.02 last 18.87) - open/close 561 (time 1787789596.34 total 30.11 last 17.65) - open/close 759 (time 1787789607.35 total 41.11 last 17.99) - open/close 938 (time 1787789617.88 total 51.65 last 16.99) - open/close 1103 (time 1787789628.67 total 62.44 last 15.29) - open/close 1302 (time 1787789639.35 total 73.12 last 18.64) - open/close 1487 (time 1787789649.35 total 83.12 last 18.49) - open/close 1713 (time 1787789660.28 total 94.04 last 20.69) - open/close 1912 (time 1787789670.28 total 104.04 last 19.90) - open/close 2155 (time 1787789680.29 total 114.05 last 24.27) - open/close 2351 (time 1787789690.74 total 124.51 last 18.75) - open/close 2536 (time 1787789700.75 total 134.51 last 18.50) - open/close 2758 (time 1787789710.77 total 144.54 last 22.14) - open/close 2943 (time 1787789720.77 total 154.54 last 18.50) - open/close 3213 (time 1787789731.65 total 165.41 last 24.84) - open/close 3453 (time 1787789741.65 total 175.41 last 23.99) - open/close 3675 (time 1787789752.12 total 185.88 last 21.20) - open/close 3895 (time 1787789762.81 total 196.58 last 20.57) - open/close 4125 (time 1787789773.49 total 207.25 last 21.55) - open/close 4382 (time 1787789783.92 total 217.69 last 24.64) - open/close 4579 (time 1787789794.44 total 228.20 last 18.73) - open/close 4812 (time 1787789804.44 total 238.21 last 23.28) - open/close 5076 (time 1787789815.10 total 248.86 last 24.78) - open/close 5297 (time 1787789825.14 total 258.90 last 22.02) - open/close 5595 (time 1787789836.18 total 269.94 last 26.99) - open/close 5824 (time 1787789847.09 total 280.86 last 20.98) - open/close 5982 (time 1787789857.59 total 291.36 last 15.05) - open/close 6151 (time 1787789868.15 total 301.92 last 16.00) - open/close 6327 (time 1787789878.16 total 311.93 last 17.58) - open/close 6514 (time 1787789889.13 total 322.90 last 17.05) - open/close 6678 (time 1787789899.64 total 333.40 last 15.61) - open/close 6817 (time 1787789909.80 total 343.57 last 13.67) - open/close 6987 (time 1787789920.63 total 354.39 last 15.71) - open/close 7117 (time 1787789930.63 total 364.40 last 12.99) - open/close 7299 (time 1787789940.64 total 374.40 last 18.19) - open/close 7474 (time 1787789950.65 total 384.42 last 17.48) - open/close 7737 (time 1787789961.08 total 394.85 last 25.21) - open/close 7943 (time 1787789971.76 total 405.53 last 19.29) - open/close 8151 (time 1787789982.26 total 416.03 last 19.82) - open/close 8241 (time 1787789992.56 total 426.33 last 8.73) - open/close 8436 (time 1787790002.95 total 436.71 last 18.78) - open/close 8641 (time 1787790013.23 total 446.99 last 19.94) - open/close 8909 (time 1787790023.92 total 457.68 last 25.07) - open/close 9166 (time 1787790034.88 total 468.65 last 23.44) - open/close 9409 (time 1787790045.75 total 479.51 last 22.36) - open/close 9570 (time 1787790056.38 total 490.15 last 15.14) - open/close 9734 (time 1787790067.38 total 501.15 last 14.91) - open/close 9908 (time 1787790078.07 total 511.84 last 16.27) - open/close 10000 (time 1787790083.45 total 517.22 last 17.11) - open/close 10138 (time 1787790093.55 total 527.32 last 13.66) - open/close 10384 (time 1787790104.31 total 538.08 last 22.87) - open/close 10537 (time 1787790114.48 total 548.25 last 15.05) - open/close 10731 (time 1787790125.48 total 559.25 last 17.63) - open/close 10927 (time 1787790135.61 total 569.38 last 19.35) - open/close 11160 (time 1787790145.61 total 579.38 last 23.30) - open/close 11390 (time 1787790155.62 total 589.39 last 22.99) - open/close 11667 (time 1787790166.00 total 599.77 last 26.68) - open/close 11905 (time 1787790176.01 total 609.77 last 23.80) - open/close 12185 (time 1787790186.79 total 620.56 last 25.95) - open/close 12398 (time 1787790196.99 total 630.75 last 20.89) - open/close 12574 (time 1787790207.00 total 640.76 last 17.58) - open/close 12738 (time 1787790217.15 total 650.92 last 16.15) - open/close 12918 (time 1787790227.19 total 660.95 last 17.94) - open/close 13155 (time 1787790237.72 total 671.49 last 22.49) - open/close 13382 (time 1787790248.05 total 681.81 last 21.99) - open/close 13592 (time 1787790258.67 total 692.44 last 19.76) - open/close 13822 (time 1787790271.67 total 705.44 last 17.69) - open/close 14021 (time 1787790282.50 total 716.27 last 18.37) - open/close 14292 (time 1787790292.52 total 726.28 last 27.07) - open/close 14526 (time 1787790302.53 total 736.29 last 23.38) - open/close 14717 (time 1787790312.88 total 746.64 last 18.45) - open/close 14913 (time 1787790323.37 total 757.14 last 18.67) - open/close 15161 (time 1787790333.39 total 767.15 last 24.77) - open/close 15931 (time 1787790343.40 total 777.16 last 76.91) - open/close 16659 (time 1787790353.40 total 787.17 last 72.76) - open/close 17487 (time 1787790363.41 total 797.18 last 82.74) - open/close 18177 (time 1787790373.42 total 807.19 last 68.92) - open/close 18922 (time 1787790383.42 total 817.19 last 74.50) - open/close 19643 (time 1787790393.43 total 827.20 last 72.05) - open/close 20000 (time 1787790398.59 total 832.35 last 69.21) - open/close 20775 (time 1787790408.60 total 842.36 last 77.44) - open/close 21485 (time 1787790418.60 total 852.36 last 70.99) - open/close 22203 (time 1787790428.60 total 862.37 last 71.77) - open/close 22913 (time 1787790438.60 total 872.37 last 71.00) - open/close 23693 (time 1787790448.60 total 882.37 last 77.98) - open/close 24517 (time 1787790458.63 total 892.39 last 82.23) - open/close 25256 (time 1787790468.63 total 902.39 last 73.88) - open/close 26030 (time 1787790478.64 total 912.41 last 77.31) - open/close 26801 (time 1787790488.66 total 922.42 last 76.98) - open/close 27534 (time 1787790498.67 total 932.44 last 73.20) - open/close 28296 (time 1787790508.68 total 942.44 last 76.15) - open/close 29096 (time 1787790518.69 total 952.45 last 79.92) - open/close 29870 (time 1787790528.69 total 962.45 last 77.38) - open/close 30672 (time 1787790538.70 total 972.47 last 80.11) - open/close 31370 (time 1787790548.71 total 982.47 last 69.74) - open/close 32090 (time 1787790558.72 total 992.48 last 71.92) - open/close 32807 (time 1787790568.72 total 1002.49 last 71.69) - open/close 33665 (time 1787790578.73 total 1012.50 last 85.70) - open/close 34377 (time 1787790588.74 total 1022.51 last 71.15) - open/close 35162 (time 1787790598.74 total 1032.51 last 78.49) - open/close 35971 (time 1787790608.75 total 1042.51 last 80.85) - open/close 36763 (time 1787790618.76 total 1052.53 last 79.08) - open/close 37492 (time 1787790628.77 total 1062.54 last 72.84) - open/close 38273 (time 1787790638.77 total 1072.54 last 78.07) - open/close 38948 (time 1787790648.78 total 1082.54 last 67.49) - open/close 39702 (time 1787790658.78 total 1092.55 last 75.36) - open/close 40000 (time 1787790662.64 total 1096.40 last 77.24) - open/close 40700 (time 1787790672.65 total 1106.42 last 69.89) - open/close 41382 (time 1787790682.66 total 1116.42 last 68.17) - open/close 42014 (time 1787790692.67 total 1126.44 last 63.11) - open/close 42710 (time 1787790702.68 total 1136.45 last 69.53) - open/close 43439 (time 1787790712.70 total 1146.46 last 72.81) - open/close 44200 (time 1787790722.70 total 1156.46 last 76.09) - open/close 44926 (time 1787790732.70 total 1166.47 last 72.54) - open/close 45666 (time 1787790742.71 total 1176.48 last 73.94) - open/close 46437 (time 1787790752.72 total 1186.49 last 77.02) - open/close 47215 (time 1787790762.73 total 1196.49 last 77.77) - open/close 47953 (time 1787790772.74 total 1206.51 last 73.71) - open/close 48668 (time 1787790782.74 total 1216.51 last 71.48) - open/close 49418 (time 1787790792.74 total 1226.51 last 74.99) total: 50000 open/close in 1233.53 seconds: 40.53 ops/second - unlinked 0 (time 1787790803 ; total 0 ; last 0) - unlinked 10000 (time 1787790941 ; total 138 ; last 138) - unlinked 20000 (time 1787791080 ; total 277 ; last 139) - unlinked 30000 (time 1787791222 ; total 419 ; last 142) - unlinked 40000 (time 1787791371 ; total 568 ; last 149) total: 50000 unlinks in 704 seconds: 71.022728 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2