== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:20:51 (1761240051) start mds service on oleg310-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 185 (time 1761240121.19 total 11.85 last 15.61) - open/close 426 (time 1761240131.20 total 21.86 last 24.08) - open/close 631 (time 1761240141.21 total 31.88 last 20.47) - open/close 815 (time 1761240151.65 total 42.32 last 17.62) - open/close 1044 (time 1761240161.92 total 52.58 last 22.32) - open/close 1228 (time 1761240171.92 total 62.58 last 18.39) - open/close 1430 (time 1761240181.93 total 72.59 last 20.17) - open/close 1660 (time 1761240191.97 total 82.63 last 22.92) - open/close 1891 (time 1761240201.97 total 92.63 last 23.09) - open/close 2073 (time 1761240212.32 total 102.98 last 17.59) - open/close 2290 (time 1761240223.07 total 113.74 last 20.18) - open/close 2505 (time 1761240233.78 total 124.44 last 20.09) - open/close 2646 (time 1761240243.94 total 134.60 last 13.87) - open/close 2785 (time 1761240253.95 total 144.61 last 13.89) - open/close 2915 (time 1761240264.04 total 154.70 last 12.89) - open/close 3071 (time 1761240274.54 total 165.20 last 14.86) - open/close 3251 (time 1761240285.48 total 176.15 last 16.44) - open/close 3441 (time 1761240295.78 total 186.44 last 18.46) - open/close 3662 (time 1761240306.80 total 197.46 last 20.05) - open/close 3882 (time 1761240316.96 total 207.63 last 21.64) - open/close 4079 (time 1761240327.58 total 218.25 last 18.55) - open/close 4277 (time 1761240338.11 total 228.77 last 18.81) - open/close 4517 (time 1761240348.59 total 239.25 last 22.90) - open/close 4771 (time 1761240359.14 total 249.80 last 24.08) - open/close 4903 (time 1761240369.44 total 260.10 last 12.82) - open/close 5090 (time 1761240379.72 total 270.39 last 18.18) - open/close 5309 (time 1761240390.63 total 281.29 last 20.08) - open/close 5458 (time 1761240401.32 total 291.99 last 13.94) - open/close 5565 (time 1761240412.19 total 302.86 last 9.84) - open/close 5664 (time 1761240422.95 total 313.61 last 9.20) - open/close 5802 (time 1761240433.44 total 324.10 last 13.16) - open/close 5983 (time 1761240444.39 total 335.05 last 16.54) - open/close 6134 (time 1761240454.50 total 345.16 last 14.93) - open/close 6316 (time 1761240465.31 total 355.97 last 16.83) - open/close 6500 (time 1761240475.36 total 366.02 last 18.32) - open/close 6662 (time 1761240485.99 total 376.65 last 15.24) - open/close 6797 (time 1761240496.22 total 386.88 last 13.19) - open/close 7015 (time 1761240506.87 total 397.53 last 20.48) - open/close 7119 (time 1761240516.88 total 407.54 last 10.38) - open/close 7233 (time 1761240526.95 total 417.61 last 11.32) - open/close 7413 (time 1761240536.95 total 427.61 last 18.00) - open/close 7620 (time 1761240546.98 total 437.64 last 20.65) - open/close 7803 (time 1761240557.34 total 448.01 last 17.65) - open/close 7967 (time 1761240567.47 total 458.13 last 16.19) - open/close 8144 (time 1761240577.49 total 468.15 last 17.67) - open/close 8308 (time 1761240587.50 total 478.16 last 16.39) - open/close 8474 (time 1761240598.57 total 489.23 last 14.99) - open/close 8636 (time 1761240608.58 total 499.24 last 16.19) - open/close 8826 (time 1761240618.92 total 509.58 last 18.38) - open/close 8989 (time 1761240628.92 total 519.58 last 16.29) - open/close 9171 (time 1761240639.08 total 529.75 last 17.91) - open/close 9408 (time 1761240649.09 total 539.75 last 23.69) - open/close 9665 (time 1761240660.09 total 550.75 last 23.36) - open/close 9866 (time 1761240670.50 total 561.17 last 19.29) - open/close 10000 (time 1761240677.98 total 568.64 last 17.92) - open/close 10206 (time 1761240688.32 total 578.98 last 19.93) - open/close 10384 (time 1761240698.34 total 589.00 last 17.76) - open/close 10595 (time 1761240708.95 total 599.62 last 19.88) - open/close 10725 (time 1761240719.84 total 610.50 last 11.95) - open/close 10789 (time 1761240730.15 total 620.81 last 6.21) - open/close 10919 (time 1761240740.32 total 630.98 last 12.78) - open/close 11025 (time 1761240752.24 total 642.91 last 8.89) - open/close 11177 (time 1761240762.66 total 653.32 last 14.60) - open/close 11279 (time 1761240773.48 total 664.14 last 9.43) - open/close 11484 (time 1761240784.04 total 674.71 last 19.40) - open/close 11714 (time 1761240794.05 total 684.71 last 22.99) - open/close 11892 (time 1761240804.14 total 694.80 last 17.64) - open/close 12085 (time 1761240814.65 total 705.31 last 18.37) - open/close 12884 (time 1761240824.65 total 715.32 last 79.86) - open/close 13689 (time 1761240834.66 total 725.32 last 80.44) - open/close 14460 (time 1761240844.66 total 735.33 last 77.07) - open/close 15231 (time 1761240854.68 total 745.34 last 76.99) - open/close 16005 (time 1761240864.69 total 755.35 last 77.32) - open/close 16804 (time 1761240874.70 total 765.37 last 79.80) - open/close 17554 (time 1761240884.71 total 775.37 last 74.93) - open/close 18348 (time 1761240894.72 total 785.38 last 79.36) - open/close 19189 (time 1761240904.72 total 795.39 last 84.03) - open/close 20000 (time 1761240913.92 total 804.58 last 88.18) - open/close 20741 (time 1761240923.93 total 814.59 last 74.05) - open/close 21488 (time 1761240933.94 total 824.60 last 74.63) - open/close 22352 (time 1761240943.95 total 834.61 last 86.32) - open/close 23197 (time 1761240953.95 total 844.61 last 84.48) - open/close 23994 (time 1761240963.96 total 854.62 last 79.66) - open/close 24730 (time 1761240973.96 total 864.62 last 73.56) - open/close 25449 (time 1761240983.96 total 874.63 last 71.89) - open/close 26229 (time 1761240993.97 total 884.63 last 77.95) - open/close 27093 (time 1761241003.98 total 894.64 last 86.35) - open/close 27948 (time 1761241013.98 total 904.65 last 85.44) - open/close 28782 (time 1761241023.99 total 914.65 last 83.35) - open/close 29616 (time 1761241034.00 total 924.66 last 83.32) - open/close 30000 (time 1761241038.84 total 929.50 last 79.36) - open/close 30727 (time 1761241048.85 total 939.51 last 72.62) - open/close 31497 (time 1761241058.85 total 949.52 last 76.96) - open/close 32261 (time 1761241068.86 total 959.53 last 76.32) - open/close 33034 (time 1761241078.88 total 969.54 last 77.16) - open/close 33790 (time 1761241088.92 total 979.58 last 75.30) - open/close 34509 (time 1761241098.93 total 989.59 last 71.85) - open/close 35144 (time 1761241108.93 total 999.60 last 63.45) - open/close 35907 (time 1761241118.94 total 1009.61 last 76.23) - open/close 36590 (time 1761241128.95 total 1019.61 last 68.27) - open/close 37301 (time 1761241138.96 total 1029.62 last 71.04) - open/close 38056 (time 1761241148.96 total 1039.62 last 75.46) - open/close 38739 (time 1761241158.98 total 1049.64 last 68.17) - open/close 39470 (time 1761241168.98 total 1059.64 last 73.10) - open/close 40000 (time 1761241175.67 total 1066.34 last 79.17) - open/close 40722 (time 1761241185.68 total 1076.34 last 72.18) - open/close 41411 (time 1761241195.69 total 1086.35 last 68.81) - open/close 42141 (time 1761241205.69 total 1096.36 last 72.99) - open/close 42911 (time 1761241215.70 total 1106.37 last 76.92) - open/close 43665 (time 1761241225.70 total 1116.37 last 75.39) - open/close 44421 (time 1761241235.71 total 1126.37 last 75.58) - open/close 45156 (time 1761241245.71 total 1136.38 last 73.47) - open/close 45823 (time 1761241255.72 total 1146.38 last 66.64) - open/close 46565 (time 1761241265.72 total 1156.39 last 74.19) - open/close 47268 (time 1761241275.72 total 1166.39 last 70.29) - open/close 48055 (time 1761241285.73 total 1176.39 last 78.67) - open/close 48871 (time 1761241295.73 total 1186.39 last 81.57) - open/close 49653 (time 1761241305.74 total 1196.41 last 78.10) total: 50000 open/close in 1201.19 seconds: 41.63 ops/second - unlinked 0 (time 1761241314 ; total 0 ; last 0) - unlinked 10000 (time 1761241451 ; total 137 ; last 137) - unlinked 20000 (time 1761241588 ; total 274 ; last 137) - unlinked 30000 (time 1761241714 ; total 400 ; last 126) - unlinked 40000 (time 1761241842 ; total 528 ; last 128) total: 50000 unlinks in 657 seconds: 76.103500 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg310-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2