== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:22:13 (1763342533) start mds service on oleg332-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg332-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 242 (time 1763342627.54 total 10.51 last 23.04) - open/close 455 (time 1763342637.73 total 20.69 last 20.91) - open/close 725 (time 1763342647.74 total 30.70 last 26.96) - open/close 947 (time 1763342657.88 total 40.84 last 21.89) - open/close 1177 (time 1763342668.08 total 51.04 last 22.55) - open/close 1399 (time 1763342679.06 total 62.02 last 20.21) - open/close 1584 (time 1763342689.50 total 72.46 last 17.73) - open/close 1820 (time 1763342699.50 total 82.46 last 23.59) - open/close 2042 (time 1763342709.51 total 92.47 last 22.18) - open/close 2289 (time 1763342720.36 total 103.33 last 22.76) - open/close 2524 (time 1763342730.87 total 113.84 last 22.35) - open/close 2817 (time 1763342741.52 total 124.48 last 27.52) - open/close 3054 (time 1763342752.10 total 135.06 last 22.40) - open/close 3290 (time 1763342763.10 total 146.06 last 21.46) - open/close 3513 (time 1763342773.12 total 156.08 last 22.25) - open/close 3640 (time 1763342784.02 total 166.99 last 11.65) - open/close 3866 (time 1763342794.97 total 177.93 last 20.64) - open/close 4096 (time 1763342805.09 total 188.05 last 22.73) - open/close 4319 (time 1763342815.09 total 198.06 last 22.29) - open/close 4553 (time 1763342825.50 total 208.47 last 22.47) - open/close 4761 (time 1763342835.51 total 218.48 last 20.78) - open/close 5028 (time 1763342845.52 total 228.49 last 26.68) - open/close 5302 (time 1763342855.53 total 238.49 last 27.38) - open/close 5570 (time 1763342865.53 total 248.49 last 26.79) - open/close 5715 (time 1763342876.19 total 259.15 last 13.61) - open/close 5968 (time 1763342886.69 total 269.66 last 24.08) - open/close 6172 (time 1763342897.56 total 280.53 last 18.77) - open/close 6356 (time 1763342907.58 total 290.54 last 18.37) - open/close 6612 (time 1763342917.91 total 300.88 last 24.77) - open/close 6847 (time 1763342927.92 total 310.89 last 23.48) - open/close 7019 (time 1763342937.93 total 320.89 last 17.19) - open/close 7243 (time 1763342948.96 total 331.93 last 20.30) - open/close 7520 (time 1763342958.97 total 341.94 last 27.67) - open/close 7733 (time 1763342969.63 total 352.59 last 19.99) - open/close 7969 (time 1763342979.64 total 362.60 last 23.57) - open/close 8184 (time 1763342989.66 total 372.63 last 21.45) - open/close 8413 (time 1763343000.28 total 383.25 last 21.57) - open/close 8634 (time 1763343010.45 total 393.41 last 21.73) - open/close 8900 (time 1763343021.28 total 404.25 last 24.55) - open/close 9160 (time 1763343032.12 total 415.08 last 24.00) - open/close 9373 (time 1763343042.13 total 425.10 last 21.27) - open/close 9566 (time 1763343052.56 total 435.53 last 18.50) - open/close 9796 (time 1763343062.57 total 445.53 last 22.99) - open/close 10000 (time 1763343071.16 total 454.13 last 23.73) - open/close 10208 (time 1763343081.18 total 464.14 last 20.78) - open/close 10439 (time 1763343092.18 total 475.15 last 20.99) - open/close 10616 (time 1763343102.20 total 485.16 last 17.68) - open/close 10827 (time 1763343112.86 total 495.82 last 19.79) - open/close 11032 (time 1763343123.61 total 506.57 last 19.07) - open/close 11319 (time 1763343133.62 total 516.58 last 28.68) - open/close 11451 (time 1763343143.72 total 526.68 last 13.06) - open/close 11587 (time 1763343153.73 total 536.70 last 13.58) - open/close 11785 (time 1763343164.76 total 547.73 last 17.95) - open/close 11998 (time 1763343174.76 total 557.73 last 21.30) - open/close 12245 (time 1763343184.91 total 567.88 last 24.34) - open/close 12461 (time 1763343195.75 total 578.71 last 19.94) - open/close 12690 (time 1763343206.64 total 589.60 last 21.03) - open/close 12931 (time 1763343217.64 total 600.61 last 21.90) - open/close 13169 (time 1763343228.31 total 611.28 last 22.31) - open/close 13437 (time 1763343239.41 total 622.37 last 24.15) - open/close 13622 (time 1763343253.44 total 636.40 last 13.19) - open/close 13849 (time 1763343264.10 total 647.06 last 21.30) - open/close 14083 (time 1763343274.26 total 657.23 last 23.01) - open/close 14326 (time 1763343284.39 total 667.36 last 24.00) - open/close 14557 (time 1763343294.41 total 677.37 last 23.06) - open/close 14772 (time 1763343304.43 total 687.39 last 21.46) - open/close 14989 (time 1763343315.18 total 698.14 last 20.18) - open/close 15159 (time 1763343325.19 total 708.15 last 16.98) - open/close 15335 (time 1763343335.20 total 718.16 last 17.59) - open/close 15509 (time 1763343346.01 total 728.98 last 16.09) - open/close 15675 (time 1763343356.04 total 739.00 last 16.56) - open/close 15906 (time 1763343367.02 total 749.99 last 21.02) - open/close 16164 (time 1763343377.95 total 760.92 last 23.61) - open/close 16383 (time 1763343387.96 total 770.93 last 21.87) - open/close 16605 (time 1763343398.94 total 781.91 last 20.23) - open/close 16803 (time 1763343409.43 total 792.39 last 18.88) - open/close 16997 (time 1763343419.81 total 802.77 last 18.69) - open/close 17211 (time 1763343430.62 total 813.59 last 19.80) - open/close 17590 (time 1763343440.63 total 823.59 last 37.88) - open/close 18422 (time 1763343450.63 total 833.59 last 83.18) - open/close 19249 (time 1763343460.63 total 843.60 last 82.67) - open/close 20000 (time 1763343468.68 total 851.65 last 93.30) - open/close 20910 (time 1763343478.68 total 861.65 last 91.00) - open/close 21798 (time 1763343488.69 total 871.65 last 88.73) - open/close 22560 (time 1763343498.69 total 881.66 last 76.18) - open/close 23432 (time 1763343508.69 total 891.66 last 87.19) - open/close 24201 (time 1763343518.70 total 901.66 last 76.86) - open/close 25004 (time 1763343528.70 total 911.67 last 80.28) - open/close 25832 (time 1763343538.70 total 921.67 last 82.79) - open/close 26659 (time 1763343548.70 total 931.67 last 82.68) - open/close 27529 (time 1763343558.71 total 941.68 last 86.94) - open/close 28365 (time 1763343568.71 total 951.68 last 83.60) - open/close 29180 (time 1763343578.72 total 961.68 last 81.46) - open/close 30000 (time 1763343588.34 total 971.31 last 85.19) - open/close 30893 (time 1763343598.35 total 981.32 last 89.19) - open/close 31755 (time 1763343608.36 total 991.32 last 86.18) - open/close 32493 (time 1763343618.36 total 1001.33 last 73.77) - open/close 33184 (time 1763343628.37 total 1011.33 last 69.06) - open/close 33878 (time 1763343638.37 total 1021.34 last 69.36) - open/close 34565 (time 1763343648.41 total 1031.38 last 68.44) - open/close 35342 (time 1763343658.41 total 1041.38 last 77.69) - open/close 36188 (time 1763343668.42 total 1051.38 last 84.58) - open/close 36967 (time 1763343678.42 total 1061.38 last 77.86) - open/close 37770 (time 1763343688.42 total 1071.39 last 80.27) - open/close 38612 (time 1763343698.44 total 1081.40 last 84.08) - open/close 39417 (time 1763343708.44 total 1091.41 last 80.44) - open/close 40000 (time 1763343714.84 total 1097.80 last 91.16) - open/close 40824 (time 1763343724.85 total 1107.82 last 82.30) - open/close 41644 (time 1763343734.86 total 1117.82 last 81.94) - open/close 42480 (time 1763343744.87 total 1127.83 last 83.53) - open/close 43382 (time 1763343754.87 total 1137.83 last 90.19) - open/close 44281 (time 1763343764.87 total 1147.83 last 89.88) - open/close 45158 (time 1763343774.89 total 1157.85 last 87.55) - open/close 46048 (time 1763343784.93 total 1167.90 last 88.58) - open/close 46926 (time 1763343794.94 total 1177.90 last 87.75) - open/close 47832 (time 1763343804.97 total 1187.94 last 90.31) - open/close 48645 (time 1763343814.99 total 1197.95 last 81.16) - open/close 49506 (time 1763343825.01 total 1207.97 last 85.92) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1209.61 seconds: 41.03 ops/second - unlinked 0 (time 1763343829 ; total 0 ; last 0) - unlinked 10000 (time 1763343951 ; total 122 ; last 122) - unlinked 20000 (time 1763344075 ; total 246 ; last 124) - unlinked 30000 (time 1763344202 ; total 373 ; last 127) - unlinked 40000 (time 1763344324 ; total 495 ; last 122) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 618 seconds: 80.310677 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2