== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:04:14 (1765105454) start mds service on oleg118-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 262 (time 1765105517.52 total 10.02 last 26.14) - open/close 544 (time 1765105527.76 total 20.26 last 27.54) - open/close 833 (time 1765105537.93 total 30.43 last 28.41) - open/close 1046 (time 1765105548.63 total 41.13 last 19.92) - open/close 1270 (time 1765105559.31 total 51.81 last 20.97) - open/close 1480 (time 1765105569.93 total 62.43 last 19.77) - open/close 1687 (time 1765105580.44 total 72.93 last 19.71) - open/close 1843 (time 1765105590.61 total 83.11 last 15.33) - open/close 2085 (time 1765105601.61 total 94.11 last 21.99) - open/close 2334 (time 1765105612.37 total 104.86 last 23.16) - open/close 2500 (time 1765105622.74 total 115.24 last 16.00) - open/close 2735 (time 1765105633.68 total 126.18 last 21.49) - open/close 2901 (time 1765105643.99 total 136.49 last 16.10) - open/close 3065 (time 1765105654.17 total 146.66 last 16.11) - open/close 3190 (time 1765105664.34 total 156.83 last 12.29) - open/close 3261 (time 1765105675.09 total 167.58 last 6.60) - open/close 3387 (time 1765105686.10 total 178.60 last 11.44) - open/close 3540 (time 1765105696.14 total 188.64 last 15.23) - open/close 3711 (time 1765105706.51 total 199.01 last 16.49) - open/close 3847 (time 1765105716.52 total 209.02 last 13.59) - open/close 4005 (time 1765105726.67 total 219.17 last 15.57) - open/close 4127 (time 1765105736.79 total 229.29 last 12.06) - open/close 4326 (time 1765105747.28 total 239.78 last 18.96) - open/close 4585 (time 1765105758.22 total 250.72 last 23.67) - open/close 4854 (time 1765105769.17 total 261.66 last 24.58) - open/close 4974 (time 1765105780.11 total 272.61 last 10.96) - open/close 5213 (time 1765105790.16 total 282.66 last 23.78) - open/close 5509 (time 1765105800.72 total 293.22 last 28.03) - open/close 5679 (time 1765105811.72 total 304.22 last 15.45) - open/close 6009 (time 1765105822.16 total 314.66 last 31.62) - open/close 6153 (time 1765105832.27 total 324.77 last 14.25) - open/close 6373 (time 1765105842.77 total 335.27 last 20.95) - open/close 6553 (time 1765105854.93 total 347.43 last 14.80) - open/close 6753 (time 1765105865.29 total 357.79 last 19.29) - open/close 6926 (time 1765105875.67 total 368.17 last 16.67) - open/close 7089 (time 1765105886.03 total 378.53 last 15.73) - open/close 9162 (time 1765105896.03 total 388.53 last 207.20) - open/close 10000 (time 1765105900.49 total 392.99 last 188.22) - open/close 12062 (time 1765105910.49 total 402.99 last 206.17) - open/close 14012 (time 1765105920.49 total 412.99 last 194.98) - open/close 16363 (time 1765105930.49 total 422.99 last 235.04) - open/close 18612 (time 1765105940.49 total 432.99 last 224.86) - open/close 20000 (time 1765105946.38 total 438.88 last 235.85) - open/close 22559 (time 1765105956.38 total 448.88 last 255.82) - open/close 24712 (time 1765105966.38 total 458.88 last 215.23) - open/close 26907 (time 1765105976.39 total 468.89 last 219.43) - open/close 28965 (time 1765105986.39 total 478.89 last 205.76) - open/close 30000 (time 1765105991.78 total 484.28 last 191.92) - open/close 32194 (time 1765106001.78 total 494.28 last 219.37) - open/close 34961 (time 1765106011.78 total 504.28 last 276.66) - open/close 37495 (time 1765106021.79 total 514.29 last 253.35) - open/close 40000 (time 1765106031.18 total 523.68 last 266.71) - open/close 42589 (time 1765106041.18 total 533.68 last 258.89) - open/close 45355 (time 1765106051.18 total 543.68 last 276.54) - open/close 47908 (time 1765106061.18 total 553.68 last 255.25) total: 50000 open/close in 559.83 seconds: 89.31 ops/second - unlinked 0 (time 1765106068 ; total 0 ; last 0) - unlinked 10000 (time 1765106089 ; total 21 ; last 21) - unlinked 20000 (time 1765106113 ; total 45 ; last 24) - unlinked 30000 (time 1765106135 ; total 67 ; last 22) - unlinked 40000 (time 1765106157 ; total 89 ; last 22) total: 50000 unlinks in 113 seconds: 442.477875 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2