== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:24:28 (1709789068) start mds service on oleg422-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: detected 4 online CPUs by sysfs oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg422-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg422-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock oleg422-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 332 (time 1709789122.17 total 10.00 last 33.19) - open/close 620 (time 1709789132.82 total 20.65 last 27.05) - open/close 900 (time 1709789142.92 total 30.75 last 27.72) - open/close 1257 (time 1709789153.47 total 41.30 last 33.83) - open/close 1600 (time 1709789163.73 total 51.56 last 33.44) - open/close 1885 (time 1709789174.52 total 62.35 last 26.40) - open/close 2279 (time 1709789184.53 total 72.36 last 39.38) - open/close 2528 (time 1709789194.82 total 82.65 last 24.19) - open/close 2804 (time 1709789205.06 total 92.89 last 26.95) - open/close 3067 (time 1709789215.46 total 103.29 last 25.30) - open/close 3229 (time 1709789225.96 total 113.79 last 15.43) - open/close 3689 (time 1709789236.79 total 124.62 last 42.48) - open/close 3979 (time 1709789252.71 total 140.54 last 18.21) - open/close 4257 (time 1709789263.34 total 151.17 last 26.16) - open/close 4497 (time 1709789273.34 total 161.17 last 23.99) - open/close 4779 (time 1709789284.26 total 172.09 last 25.83) - open/close 4859 (time 1709789295.21 total 183.04 last 7.30) - open/close 5134 (time 1709789305.22 total 193.05 last 27.49) - open/close 5414 (time 1709789316.20 total 204.03 last 25.49) - open/close 5733 (time 1709789326.53 total 214.36 last 30.89) - open/close 5967 (time 1709789337.28 total 225.11 last 21.77) - open/close 6168 (time 1709789347.81 total 235.64 last 19.08) - open/close 6473 (time 1709789358.54 total 246.37 last 28.42) - open/close 6770 (time 1709789368.96 total 256.80 last 28.50) - open/close 7018 (time 1709789378.97 total 266.80 last 24.80) - open/close 7220 (time 1709789389.26 total 277.09 last 19.62) - open/close 7392 (time 1709789399.48 total 287.31 last 16.83) - open/close 7613 (time 1709789412.40 total 300.23 last 17.10) - open/close 7811 (time 1709789423.09 total 310.92 last 18.54) - open/close 8044 (time 1709789434.05 total 321.88 last 21.25) - open/close 8288 (time 1709789445.07 total 332.90 last 22.14) - open/close 9132 (time 1709789455.07 total 342.90 last 84.37) - open/close 9882 (time 1709789465.09 total 352.92 last 74.91) - open/close 10573 (time 1709789475.09 total 362.92 last 69.05) - open/close 11219 (time 1709789485.11 total 372.94 last 64.53) - open/close 11894 (time 1709789495.12 total 382.95 last 67.42) - open/close 12520 (time 1709789505.12 total 392.95 last 62.56) - open/close 13093 (time 1709789515.13 total 402.96 last 57.25) - open/close 13666 (time 1709789525.14 total 412.97 last 57.27) - open/close 14222 (time 1709789535.15 total 422.98 last 55.55) - open/close 14743 (time 1709789545.15 total 432.98 last 52.09) - open/close 15282 (time 1709789555.16 total 442.99 last 53.84) - open/close 15799 (time 1709789565.18 total 453.01 last 51.59) - open/close 16278 (time 1709789575.21 total 463.04 last 47.78) - open/close 16764 (time 1709789585.22 total 473.05 last 48.54) - open/close 17243 (time 1709789595.22 total 483.06 last 47.87) - open/close 17739 (time 1709789605.23 total 493.06 last 49.57) - open/close 18207 (time 1709789615.24 total 503.07 last 46.75) - open/close 18653 (time 1709789625.25 total 513.08 last 44.54) - open/close 19069 (time 1709789635.26 total 523.09 last 41.56) - open/close 19501 (time 1709789645.26 total 533.09 last 43.20) - open/close 19931 (time 1709789655.28 total 543.11 last 42.93) - open/close 20351 (time 1709789665.28 total 553.11 last 41.98) - open/close 20772 (time 1709789675.29 total 563.12 last 42.06) - open/close 21191 (time 1709789685.31 total 573.14 last 41.83) - open/close 21620 (time 1709789695.31 total 583.14 last 42.89) - open/close 22049 (time 1709789705.33 total 593.16 last 42.82) - open/close 22515 (time 1709789715.34 total 603.17 last 46.54) - open/close 22947 (time 1709789725.35 total 613.18 last 43.17) - open/close 23376 (time 1709789735.35 total 623.19 last 42.88) - open/close 23771 (time 1709789745.38 total 633.21 last 39.39) - open/close 24150 (time 1709789755.40 total 643.23 last 37.83) - open/close 24539 (time 1709789765.40 total 653.23 last 38.89) - open/close 24914 (time 1709789775.44 total 663.27 last 37.38) - open/close 25272 (time 1709789785.45 total 673.28 last 35.74) - open/close 25635 (time 1709789795.46 total 683.29 last 36.27) - open/close 26000 (time 1709789805.47 total 693.30 last 36.47) - open/close 26374 (time 1709789815.49 total 703.32 last 37.33) - open/close 26741 (time 1709789825.51 total 713.34 last 36.61) - open/close 27103 (time 1709789835.52 total 723.35 last 36.16) - open/close 27452 (time 1709789845.54 total 733.37 last 34.82) - open/close 27815 (time 1709789855.56 total 743.39 last 36.25) - open/close 28171 (time 1709789865.57 total 753.40 last 35.57) - open/close 28500 (time 1709789875.59 total 763.42 last 32.84) - open/close 28852 (time 1709789885.60 total 773.43 last 35.15) - open/close 29215 (time 1709789895.61 total 783.44 last 36.27) - open/close 29565 (time 1709789905.61 total 793.44 last 34.99) - open/close 29888 (time 1709789915.63 total 803.46 last 32.23) - open/close 30000 (time 1709789918.78 total 806.61 last 35.58) - open/close 30357 (time 1709789928.78 total 816.61 last 35.70) - open/close 30717 (time 1709789938.79 total 826.62 last 35.99) - open/close 31052 (time 1709789948.80 total 836.63 last 33.46) - open/close 31380 (time 1709789958.82 total 846.65 last 32.71) - open/close 31744 (time 1709789968.82 total 856.65 last 36.40) - open/close 32123 (time 1709789978.84 total 866.67 last 37.85) - open/close 32509 (time 1709789988.84 total 876.67 last 38.58) - open/close 32896 (time 1709789998.85 total 886.68 last 38.68) - open/close 33265 (time 1709790008.85 total 896.68 last 36.88) - open/close 33634 (time 1709790018.87 total 906.70 last 36.84) - open/close 34007 (time 1709790028.89 total 916.72 last 37.23) - open/close 34368 (time 1709790038.89 total 926.72 last 36.10) - open/close 34712 (time 1709790048.89 total 936.72 last 34.39) - open/close 35059 (time 1709790058.89 total 946.72 last 34.69) - open/close 35378 (time 1709790068.90 total 956.73 last 31.86) - open/close 35716 (time 1709790078.91 total 966.74 last 33.79) - open/close 36059 (time 1709790088.91 total 976.74 last 34.30) - open/close 36411 (time 1709790098.92 total 986.75 last 35.16) - open/close 36743 (time 1709790108.94 total 996.77 last 33.13) - open/close 37034 (time 1709790118.97 total 1006.80 last 29.00) - open/close 37350 (time 1709790128.99 total 1016.82 last 31.56) - open/close 37699 (time 1709790139.00 total 1026.83 last 34.86) - open/close 38034 (time 1709790149.02 total 1036.85 last 33.41) - open/close 38358 (time 1709790159.04 total 1046.87 last 32.36) - open/close 38689 (time 1709790169.05 total 1056.88 last 33.07) - open/close 38992 (time 1709790179.06 total 1066.89 last 30.25) - open/close 39384 (time 1709790189.07 total 1076.90 last 39.17) - open/close 39764 (time 1709790199.10 total 1086.93 last 37.89) - open/close 40000 (time 1709790205.23 total 1093.06 last 38.49) - open/close 40402 (time 1709790215.26 total 1103.09 last 40.09) - open/close 40817 (time 1709790225.28 total 1113.11 last 41.40) - open/close 41244 (time 1709790235.28 total 1123.11 last 42.69) - open/close 41653 (time 1709790245.30 total 1133.13 last 40.85) - open/close 42069 (time 1709790255.30 total 1143.14 last 41.57) - open/close 42487 (time 1709790265.33 total 1153.16 last 41.70) - open/close 42898 (time 1709790275.34 total 1163.17 last 41.04) - open/close 43321 (time 1709790285.36 total 1173.19 last 42.21) - open/close 43779 (time 1709790295.37 total 1183.20 last 45.77) - open/close 44205 (time 1709790305.39 total 1193.22 last 42.52) - open/close 44670 (time 1709790315.40 total 1203.23 last 46.44) - open/close 45168 (time 1709790325.41 total 1213.24 last 49.78) - open/close 45705 (time 1709790335.41 total 1223.24 last 53.69) - open/close 46225 (time 1709790345.42 total 1233.25 last 51.93) - open/close 46748 (time 1709790355.43 total 1243.26 last 52.25) - open/close 47397 (time 1709790365.44 total 1253.27 last 64.83) - open/close 48033 (time 1709790375.45 total 1263.28 last 63.58) - open/close 48592 (time 1709790385.45 total 1273.28 last 55.87) - open/close 49092 (time 1709790395.45 total 1283.28 last 49.99) - open/close 49619 (time 1709790405.47 total 1293.30 last 52.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1293.51 seconds: 38.37 ops/second - unlinked 0 (time 1709790406 ; total 0 ; last 0) - unlinked 10000 (time 1709790667 ; total 261 ; last 261) - unlinked 20000 (time 1709790921 ; total 515 ; last 254) - unlinked 30000 (time 1709791121 ; total 715 ; last 200) - unlinked 40000 (time 1709791212 ; total 806 ; last 91) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 841 seconds: 59.015457 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server unloading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2