== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:06:21 (1769756781) start mds service on oleg157-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: Force libcfs to create 2 CPU partitions oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 117 (time 1769756874.91 total 10.00 last 11.70) - open/close 371 (time 1769756884.92 total 20.01 last 25.36) - open/close 538 (time 1769756895.73 total 30.82 last 15.45) - open/close 793 (time 1769756906.58 total 41.68 last 23.49) - open/close 982 (time 1769756916.95 total 52.04 last 18.23) - open/close 1106 (time 1769756926.98 total 62.07 last 12.37) - open/close 1289 (time 1769756937.62 total 72.71 last 17.20) - open/close 1540 (time 1769756947.63 total 82.72 last 25.07) - open/close 1596 (time 1769756958.06 total 93.15 last 5.37) - open/close 1761 (time 1769756968.07 total 103.17 last 16.48) - open/close 1885 (time 1769756978.08 total 113.17 last 12.39) - open/close 1961 (time 1769756991.00 total 126.09 last 5.88) - open/close 2107 (time 1769757001.01 total 136.10 last 14.59) - open/close 2332 (time 1769757011.02 total 146.11 last 22.48) - open/close 2507 (time 1769757021.65 total 156.74 last 16.46) - open/close 2725 (time 1769757033.13 total 168.22 last 18.98) - open/close 2901 (time 1769757043.15 total 178.24 last 17.56) - open/close 3111 (time 1769757053.65 total 188.74 last 20.00) - open/close 3241 (time 1769757063.74 total 198.83 last 12.88) - open/close 3416 (time 1769757074.57 total 209.67 last 16.16) - open/close 3543 (time 1769757084.58 total 219.68 last 12.69) - open/close 3745 (time 1769757094.68 total 229.78 last 20.00) - open/close 3920 (time 1769757105.38 total 240.48 last 16.36) - open/close 4156 (time 1769757115.40 total 250.49 last 23.56) - open/close 4254 (time 1769757126.08 total 261.18 last 9.17) - open/close 4445 (time 1769757136.92 total 272.01 last 17.63) - open/close 4724 (time 1769757147.91 total 283.00 last 25.38) - open/close 4858 (time 1769757157.97 total 293.06 last 13.33) - open/close 4937 (time 1769757170.39 total 305.49 last 6.36) - open/close 5033 (time 1769757181.26 total 316.36 last 8.83) - open/close 5238 (time 1769757191.57 total 326.66 last 19.89) - open/close 5441 (time 1769757201.62 total 336.72 last 20.19) - open/close 5637 (time 1769757212.26 total 347.35 last 18.43) - open/close 5869 (time 1769757222.27 total 357.36 last 23.18) - open/close 6112 (time 1769757232.28 total 367.37 last 24.28) - open/close 6266 (time 1769757242.28 total 377.38 last 15.39) - open/close 6433 (time 1769757253.25 total 388.34 last 15.23) - open/close 6586 (time 1769757266.01 total 401.10 last 11.99) - open/close 6726 (time 1769757276.61 total 411.70 last 13.21) - open/close 6939 (time 1769757287.25 total 422.35 last 20.01) - open/close 7104 (time 1769757298.35 total 433.44 last 14.87) - open/close 7201 (time 1769757309.32 total 444.41 last 8.84) - open/close 7294 (time 1769757322.06 total 457.16 last 7.30) - open/close 7342 (time 1769757332.68 total 467.78 last 4.52) - open/close 7587 (time 1769757343.34 total 478.43 last 22.99) - open/close 7765 (time 1769757353.35 total 488.44 last 17.78) - open/close 7889 (time 1769757363.39 total 498.49 last 12.35) - open/close 8134 (time 1769757373.40 total 508.49 last 24.49) - open/close 8381 (time 1769757384.03 total 519.12 last 23.23) - open/close 8473 (time 1769757394.53 total 529.62 last 8.76) - open/close 8722 (time 1769757404.55 total 539.64 last 24.86) - open/close 8864 (time 1769757414.67 total 549.77 last 14.02) - open/close 8979 (time 1769757424.69 total 559.79 last 11.48) - open/close 9187 (time 1769757435.31 total 570.40 last 19.60) - open/close 9287 (time 1769757445.79 total 580.88 last 9.54) - open/close 9463 (time 1769757455.80 total 590.89 last 17.58) - open/close 9731 (time 1769757466.66 total 601.75 last 24.67) - open/close 9957 (time 1769757476.70 total 611.80 last 22.50) - open/close 10000 (time 1769757479.87 total 614.96 last 13.60) - open/close 10170 (time 1769757489.87 total 624.96 last 17.00) - open/close 10408 (time 1769757499.87 total 634.97 last 23.79) - open/close 10564 (time 1769757510.88 total 645.97 last 14.17) - open/close 10690 (time 1769757521.93 total 657.02 last 11.40) - open/close 10812 (time 1769757531.94 total 667.03 last 12.19) - open/close 11002 (time 1769757541.95 total 677.04 last 18.98) - open/close 11203 (time 1769757551.96 total 687.06 last 20.07) - open/close 11303 (time 1769757562.97 total 698.06 last 9.09) - open/close 11583 (time 1769757572.98 total 708.07 last 27.96) - open/close 11817 (time 1769757583.07 total 718.17 last 23.19) - open/close 11985 (time 1769757593.08 total 728.18 last 16.78) - open/close 12102 (time 1769757604.00 total 739.10 last 10.71) - open/close 12331 (time 1769757614.85 total 749.95 last 21.10) - open/close 12535 (time 1769757625.12 total 760.21 last 19.88) - open/close 12690 (time 1769757635.13 total 770.22 last 15.48) - open/close 12891 (time 1769757645.15 total 780.24 last 20.06) - open/close 13111 (time 1769757655.94 total 791.03 last 20.39) - open/close 13299 (time 1769757666.57 total 801.66 last 17.69) - open/close 13540 (time 1769757676.58 total 811.68 last 24.06) - open/close 13730 (time 1769757686.73 total 821.82 last 18.72) - open/close 13896 (time 1769757697.32 total 832.41 last 15.68) - open/close 14143 (time 1769757707.34 total 842.43 last 24.66) - open/close 14438 (time 1769757717.41 total 852.50 last 29.30) - open/close 14632 (time 1769757727.51 total 862.61 last 19.20) - open/close 14721 (time 1769757737.69 total 872.78 last 8.75) - open/close 14970 (time 1769757747.70 total 882.80 last 24.86) - open/close 15225 (time 1769757757.72 total 892.81 last 25.46) - open/close 15399 (time 1769757767.72 total 902.81 last 17.40) - open/close 15479 (time 1769757777.74 total 912.83 last 7.99) - open/close 15692 (time 1769757788.49 total 923.58 last 19.81) - open/close 15854 (time 1769757799.24 total 934.33 last 15.07) - open/close 16093 (time 1769757809.26 total 944.36 last 23.85) - open/close 16326 (time 1769757819.35 total 954.44 last 23.10) - open/close 16491 (time 1769757829.99 total 965.08 last 15.51) - open/close 16728 (time 1769757840.02 total 975.11 last 23.64) - open/close 16995 (time 1769757850.52 total 985.61 last 25.43) - open/close 17195 (time 1769757860.54 total 995.63 last 19.95) - open/close 17422 (time 1769757870.73 total 1005.82 last 22.28) - open/close 17618 (time 1769757881.68 total 1016.77 last 17.90) - open/close 17868 (time 1769757892.17 total 1027.26 last 23.83) - open/close 18560 (time 1769757902.18 total 1037.27 last 69.17) - open/close 19294 (time 1769757912.19 total 1047.28 last 73.28) - open/close 20000 (time 1769757920.83 total 1055.92 last 81.73) - open/close 20814 (time 1769757930.84 total 1065.93 last 81.31) - open/close 21518 (time 1769757940.85 total 1075.94 last 70.36) - open/close 22139 (time 1769757950.88 total 1085.97 last 61.89) - open/close 22958 (time 1769757960.89 total 1095.99 last 81.81) - open/close 23802 (time 1769757970.90 total 1105.99 last 84.33) - open/close 24394 (time 1769757980.91 total 1116.00 last 59.14) - open/close 25176 (time 1769757990.91 total 1126.01 last 78.17) - open/close 25989 (time 1769758000.94 total 1136.03 last 81.12) - open/close 26768 (time 1769758010.94 total 1146.03 last 77.89) - open/close 27529 (time 1769758020.95 total 1156.04 last 76.03) - open/close 28277 (time 1769758030.95 total 1166.04 last 74.78) - open/close 29164 (time 1769758040.95 total 1176.04 last 88.69) - open/close 30000 (time 1769758050.78 total 1185.87 last 85.06) - open/close 30834 (time 1769758060.78 total 1195.88 last 83.38) - open/close 31437 (time 1769758070.82 total 1205.92 last 60.05) - open/close 32239 (time 1769758080.83 total 1215.93 last 80.13) - open/close 33085 (time 1769758090.83 total 1225.93 last 84.58) - open/close 33865 (time 1769758100.85 total 1235.94 last 77.90) - open/close 34638 (time 1769758110.89 total 1245.98 last 77.00) - open/close 35359 (time 1769758120.90 total 1255.99 last 72.03) - open/close 36101 (time 1769758130.91 total 1266.00 last 74.12) - open/close 36961 (time 1769758140.91 total 1276.01 last 85.96) - open/close 37813 (time 1769758150.92 total 1286.01 last 85.17) - open/close 38743 (time 1769758160.92 total 1296.01 last 92.99) - open/close 39560 (time 1769758170.92 total 1306.01 last 81.66) - open/close 40000 (time 1769758176.82 total 1311.91 last 74.57) - open/close 40845 (time 1769758186.82 total 1321.92 last 84.49) - open/close 41726 (time 1769758196.83 total 1331.92 last 88.03) - open/close 42618 (time 1769758206.84 total 1341.93 last 89.12) - open/close 43495 (time 1769758216.84 total 1351.93 last 87.69) - open/close 44411 (time 1769758226.84 total 1361.94 last 91.57) - open/close 45256 (time 1769758236.85 total 1371.94 last 84.46) - open/close 46181 (time 1769758246.85 total 1381.94 last 92.47) - open/close 47000 (time 1769758256.85 total 1391.95 last 81.88) - open/close 47809 (time 1769758266.86 total 1401.95 last 80.87) - open/close 48672 (time 1769758276.87 total 1411.96 last 86.21) - open/close 49484 (time 1769758286.87 total 1421.96 last 81.19) total: 50000 open/close in 1427.69 seconds: 35.02 ops/second - unlinked 0 (time 1769758295 ; total 0 ; last 0) - unlinked 10000 (time 1769758414 ; total 119 ; last 119) - unlinked 20000 (time 1769758528 ; total 233 ; last 114) - unlinked 30000 (time 1769758608 ; total 313 ; last 80) - unlinked 40000 (time 1769758671 ; total 376 ; last 63) total: 50000 unlinks in 439 seconds: 113.895218 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg157-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2