== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:45:34 (1769078734) start mds service on oleg252-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions oleg252-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg252-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg252-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 118 (time 1769078846.26 total 12.95 last 9.11) - open/close 216 (time 1769078856.44 total 23.13 last 9.63) - open/close 344 (time 1769078868.60 total 35.29 last 10.52) - open/close 477 (time 1769078879.14 total 45.82 last 12.62) - open/close 582 (time 1769078890.02 total 56.71 last 9.64) - open/close 779 (time 1769078900.59 total 67.28 last 18.64) - open/close 894 (time 1769078910.61 total 77.30 last 11.48) - open/close 1127 (time 1769078920.65 total 87.34 last 23.20) - open/close 1219 (time 1769078930.71 total 97.40 last 9.15) - open/close 1303 (time 1769078941.39 total 108.08 last 7.86) - open/close 1409 (time 1769078952.11 total 118.80 last 9.89) - open/close 1483 (time 1769078963.03 total 129.72 last 6.77) - open/close 1617 (time 1769078974.12 total 140.81 last 12.09) - open/close 1724 (time 1769078985.19 total 151.88 last 9.67) - open/close 1890 (time 1769078995.49 total 162.18 last 16.11) - open/close 2023 (time 1769079005.50 total 172.19 last 13.29) - open/close 2113 (time 1769079015.83 total 182.52 last 8.71) - open/close 2240 (time 1769079025.97 total 192.66 last 12.52) - open/close 2380 (time 1769079036.82 total 203.51 last 12.90) - open/close 2473 (time 1769079055.67 total 222.35 last 4.94) - open/close 2602 (time 1769079065.96 total 232.65 last 12.54) - open/close 2676 (time 1769079076.07 total 242.76 last 7.32) - open/close 2736 (time 1769079086.11 total 252.80 last 5.97) - open/close 2816 (time 1769079096.42 total 263.11 last 7.76) - open/close 2959 (time 1769079106.50 total 273.19 last 14.18) - open/close 3088 (time 1769079116.52 total 283.21 last 12.88) - open/close 3221 (time 1769079127.62 total 294.31 last 11.98) - open/close 3262 (time 1769079137.99 total 304.68 last 3.95) - open/close 3372 (time 1769079148.91 total 315.60 last 10.07) - open/close 3563 (time 1769079158.94 total 325.63 last 19.05) - open/close 3688 (time 1769079169.43 total 336.12 last 11.92) - open/close 3745 (time 1769079179.44 total 346.13 last 5.70) - open/close 3805 (time 1769079190.18 total 356.87 last 5.58) - open/close 3888 (time 1769079200.49 total 367.18 last 8.05) - open/close 4084 (time 1769079210.49 total 377.18 last 19.60) - open/close 4285 (time 1769079220.74 total 387.43 last 19.62) - open/close 4430 (time 1769079231.20 total 397.89 last 13.86) - open/close 4677 (time 1769079241.21 total 407.90 last 24.67) - open/close 4868 (time 1769079251.36 total 418.05 last 18.81) - open/close 5061 (time 1769079262.49 total 429.18 last 17.35) - open/close 5306 (time 1769079272.74 total 439.43 last 23.89) - open/close 5564 (time 1769079282.74 total 449.43 last 25.79) - open/close 5691 (time 1769079292.75 total 459.44 last 12.69) - open/close 5848 (time 1769079302.76 total 469.45 last 15.68) - open/close 5915 (time 1769079312.80 total 479.49 last 6.67) - open/close 6024 (time 1769079323.73 total 490.42 last 9.97) - open/close 6180 (time 1769079333.74 total 500.43 last 15.59) - open/close 6353 (time 1769079344.48 total 511.17 last 16.10) - open/close 6476 (time 1769079356.19 total 522.88 last 10.51) - open/close 6648 (time 1769079366.20 total 532.89 last 17.18) - open/close 6824 (time 1769079376.21 total 542.90 last 17.59) - open/close 7010 (time 1769079387.06 total 553.75 last 17.14) - open/close 7250 (time 1769079397.98 total 564.67 last 21.97) - open/close 7492 (time 1769079408.56 total 575.25 last 22.88) - open/close 7731 (time 1769079418.57 total 585.26 last 23.88) - open/close 7971 (time 1769079428.58 total 595.26 last 23.98) - open/close 8229 (time 1769079438.58 total 605.27 last 25.78) - open/close 8475 (time 1769079448.99 total 615.68 last 23.63) - open/close 8705 (time 1769079459.18 total 625.87 last 22.59) - open/close 9020 (time 1769079469.18 total 635.87 last 31.49) - open/close 9276 (time 1769079479.71 total 646.40 last 24.30) - open/close 9437 (time 1769079492.01 total 658.70 last 13.09) - open/close 9645 (time 1769079502.37 total 669.06 last 20.08) - open/close 9854 (time 1769079512.38 total 679.07 last 20.88) - open/close 10000 (time 1769079518.13 total 684.82 last 25.40) - open/close 10268 (time 1769079528.13 total 694.82 last 26.78) - open/close 10471 (time 1769079538.13 total 704.82 last 20.30) - open/close 10733 (time 1769079549.15 total 715.84 last 23.78) - open/close 10967 (time 1769079559.58 total 726.27 last 22.43) - open/close 11221 (time 1769079569.59 total 736.27 last 25.40) - open/close 11446 (time 1769079579.60 total 746.29 last 22.47) - open/close 11697 (time 1769079590.37 total 757.06 last 23.31) - open/close 11960 (time 1769079600.38 total 767.07 last 26.26) - open/close 12150 (time 1769079610.40 total 777.09 last 18.96) - open/close 12228 (time 1769079620.44 total 787.13 last 7.77) - open/close 12432 (time 1769079630.51 total 797.20 last 20.27) - open/close 12708 (time 1769079640.53 total 807.22 last 27.55) - open/close 13028 (time 1769079650.85 total 817.54 last 31.01) - open/close 13245 (time 1769079660.85 total 827.54 last 21.69) - open/close 13432 (time 1769079670.86 total 837.55 last 18.69) - open/close 13660 (time 1769079681.81 total 848.50 last 20.81) - open/close 13751 (time 1769079691.89 total 858.58 last 9.03) - open/close 13970 (time 1769079701.90 total 868.59 last 21.88) - open/close 14236 (time 1769079712.61 total 879.30 last 24.83) - open/close 14440 (time 1769079722.62 total 889.31 last 20.38) - open/close 14631 (time 1769079732.62 total 899.31 last 19.09) - open/close 14754 (time 1769079742.62 total 909.31 last 12.30) - open/close 15039 (time 1769079752.65 total 919.34 last 28.43) - open/close 15218 (time 1769079763.49 total 930.18 last 16.51) - open/close 15473 (time 1769079773.50 total 940.19 last 25.47) - open/close 15642 (time 1769079783.50 total 950.19 last 16.89) - open/close 15885 (time 1769079794.59 total 961.28 last 21.92) - open/close 16023 (time 1769079804.59 total 971.28 last 13.79) - open/close 16233 (time 1769079814.87 total 981.56 last 20.42) - open/close 16483 (time 1769079824.89 total 991.58 last 24.96) - open/close 16687 (time 1769079835.75 total 1002.44 last 18.79) - open/close 16816 (time 1769079845.82 total 1012.51 last 12.81) - open/close 16952 (time 1769079855.84 total 1022.53 last 13.57) - open/close 17165 (time 1769079866.34 total 1033.03 last 20.28) - open/close 17573 (time 1769079876.35 total 1043.03 last 40.80) - open/close 18338 (time 1769079886.36 total 1053.05 last 76.41) - open/close 19152 (time 1769079896.36 total 1063.05 last 81.35) - open/close 19951 (time 1769079906.37 total 1073.06 last 79.84) - open/close 20760 (time 1769079916.38 total 1083.07 last 80.83) - open/close 21674 (time 1769079926.39 total 1093.08 last 91.29) - open/close 22584 (time 1769079936.40 total 1103.09 last 90.89) - open/close 23524 (time 1769079946.41 total 1113.10 last 93.96) - open/close 24454 (time 1769079956.42 total 1123.11 last 92.88) - open/close 25246 (time 1769079966.43 total 1133.12 last 79.16) - open/close 26153 (time 1769079976.43 total 1143.12 last 90.66) - open/close 27086 (time 1769079986.43 total 1153.12 last 93.30) - open/close 27993 (time 1769079996.44 total 1163.13 last 90.58) - open/close 28938 (time 1769080006.45 total 1173.14 last 94.46) - open/close 29901 (time 1769080016.45 total 1183.14 last 96.27) - open/close 30772 (time 1769080026.45 total 1193.14 last 87.07) - open/close 31611 (time 1769080036.49 total 1203.18 last 83.62) - open/close 32503 (time 1769080046.49 total 1213.18 last 89.19) - open/close 33287 (time 1769080056.49 total 1223.18 last 78.37) - open/close 34081 (time 1769080066.50 total 1233.19 last 79.39) - open/close 35008 (time 1769080076.50 total 1243.19 last 92.66) - open/close 35814 (time 1769080086.50 total 1253.19 last 80.60) - open/close 36685 (time 1769080096.50 total 1263.19 last 87.08) - open/close 37590 (time 1769080106.50 total 1273.19 last 90.49) - open/close 38522 (time 1769080116.51 total 1283.20 last 93.18) - open/close 39465 (time 1769080126.51 total 1293.20 last 94.28) - open/close 40000 (time 1769080132.29 total 1298.98 last 92.60) - open/close 40915 (time 1769080142.30 total 1308.99 last 91.41) - open/close 41722 (time 1769080152.30 total 1318.99 last 80.64) - open/close 42589 (time 1769080162.31 total 1329.00 last 86.63) - open/close 43465 (time 1769080172.32 total 1339.01 last 87.56) - open/close 44388 (time 1769080182.32 total 1349.01 last 92.27) - open/close 45247 (time 1769080192.32 total 1359.01 last 85.89) - open/close 46109 (time 1769080202.33 total 1369.02 last 86.13) - open/close 46979 (time 1769080212.33 total 1379.02 last 86.95) - open/close 47843 (time 1769080222.34 total 1389.03 last 86.38) - open/close 48744 (time 1769080232.34 total 1399.03 last 90.09) - open/close 49598 (time 1769080242.34 total 1409.03 last 85.35) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1409.48 seconds: 35.21 ops/second - unlinked 0 (time 1769080246 ; total 0 ; last 0) - unlinked 10000 (time 1769080347 ; total 101 ; last 101) - unlinked 20000 (time 1769080432 ; total 186 ; last 85) - unlinked 30000 (time 1769080484 ; total 238 ; last 52) - unlinked 40000 (time 1769080538 ; total 292 ; last 54) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 342 seconds: 145.122803 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2