== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:30:26 (1760128226) Stopping clients: oleg431-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg431-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg431-server oleg431-server: oleg431-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg431-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg431-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 901 (time 1760128400.90 total 10.02 last 89.95) - open/close 1825 (time 1760128410.91 total 20.03 last 92.32) - open/close 2952 (time 1760128420.91 total 30.03 last 112.68) - open/close 3927 (time 1760128430.92 total 40.03 last 97.47) total: 4500 open/close in 45.51 seconds: 98.88 ops/second - /unlink 1694 (time 1760128449.53 total 10.01 last 169.22) - /unlink 3566 (time 1760128459.54 total 20.02 last 187.01) total: 4500 /unlink in 24.05 seconds: 187.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 818 (time 1760128476.82 total 10.01 last 81.69) - open/close 1762 (time 1760128486.83 total 20.02 last 94.31) - open/close 2789 (time 1760128496.83 total 30.02 last 102.69) - open/close 3723 (time 1760128506.85 total 40.04 last 93.29) total: 4500 open/close in 47.37 seconds: 95.00 ops/second - /unlink 1792 (time 1760128526.81 total 10.00 last 179.15) - /unlink 3335 (time 1760128536.81 total 20.00 last 154.30) total: 4500 /unlink in 27.36 seconds: 164.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 928 (time 1760128557.26 total 10.01 last 92.71) - open/close 1800 (time 1760128567.26 total 20.01 last 87.18) - open/close 2731 (time 1760128577.26 total 30.02 last 93.06) - open/close 3694 (time 1760128587.26 total 40.02 last 96.29) total: 4500 open/close in 48.34 seconds: 93.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 2134 (time 1760128607.80 total 10.00 last 213.33) - /unlink 3964 (time 1760128617.80 total 20.00 last 182.98) total: 4500 /unlink in 22.47 seconds: 200.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 1081 (time 1760128632.46 total 10.02 last 107.90) - open/close 2203 (time 1760128642.47 total 20.03 last 112.08) - open/close 3238 (time 1760128652.47 total 30.03 last 103.49) - open/close 4336 (time 1760128662.48 total 40.04 last 109.66) total: 4500 open/close in 41.55 seconds: 108.29 ops/second - /unlink 1733 (time 1760128676.73 total 10.01 last 173.20) - /unlink 3633 (time 1760128686.73 total 20.01 last 189.97) total: 4500 /unlink in 24.56 seconds: 183.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 1196 (time 1760128703.30 total 10.01 last 119.52) - open/close 2144 (time 1760128713.32 total 20.02 last 94.65) - open/close 3102 (time 1760128723.33 total 30.03 last 95.69) - open/close 4087 (time 1760128733.33 total 40.04 last 98.48) total: 4500 open/close in 44.48 seconds: 101.18 ops/second - /unlink 1841 (time 1760128750.66 total 10.01 last 183.99) - /unlink 3577 (time 1760128760.66 total 20.01 last 173.56) total: 4500 /unlink in 25.15 seconds: 178.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 1023 (time 1760128778.65 total 10.01 last 102.19) - open/close 2027 (time 1760128788.66 total 20.03 last 100.22) - open/close 3002 (time 1760128798.67 total 30.04 last 97.43) - open/close 4212 (time 1760128808.67 total 40.04 last 121.00) total: 4500 open/close in 42.33 seconds: 106.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1827 (time 1760128822.92 total 10.00 last 182.67) - /unlink 3997 (time 1760128832.92 total 20.00 last 216.99) total: 4500 /unlink in 22.25 seconds: 202.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 1210 (time 1760128847.86 total 10.01 last 120.91) - open/close 2295 (time 1760128857.87 total 20.02 last 108.41) - open/close 3334 (time 1760128867.87 total 30.02 last 103.87) total: 4500 open/close in 39.61 seconds: 113.62 ops/second - /unlink 2123 (time 1760128889.80 total 10.00 last 212.24) total: 4500 /unlink in 19.77 seconds: 227.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 1499 (time 1760128911.44 total 10.00 last 149.87) - open/close 2765 (time 1760128921.45 total 20.01 last 126.52) - open/close 4014 (time 1760128931.45 total 30.01 last 124.90) total: 4500 open/close in 34.93 seconds: 128.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1539 (time 1760128949.00 total 10.00 last 153.85) - /unlink 3202 (time 1760128959.00 total 20.01 last 166.24) total: 4500 /unlink in 28.86 seconds: 155.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1027 (time 1760128981.23 total 10.01 last 102.63) - open/close 1930 (time 1760128991.23 total 20.01 last 90.25) - open/close 2967 (time 1760129001.23 total 30.02 last 103.66) - open/close 3799 (time 1760129011.24 total 40.02 last 83.16) total: 4500 open/close in 46.47 seconds: 96.83 ops/second - /unlink 1662 (time 1760129029.87 total 10.00 last 166.15) - /unlink 3210 (time 1760129039.88 total 20.01 last 154.73) total: 4500 /unlink in 28.46 seconds: 158.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 905 (time 1760129061.56 total 10.00 last 90.47) - open/close 1866 (time 1760129071.57 total 20.01 last 96.04) - open/close 2894 (time 1760129081.58 total 30.02 last 102.67) - open/close 3893 (time 1760129091.59 total 40.03 last 99.85) total: 4500 open/close in 45.46 seconds: 98.98 ops/second - /unlink 2275 (time 1760129109.35 total 10.00 last 227.39) - /unlink 3988 (time 1760129119.35 total 20.01 last 171.16) total: 4500 /unlink in 22.52 seconds: 199.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 945 (time 1760129134.20 total 10.01 last 94.42) - open/close 1918 (time 1760129144.21 total 20.02 last 97.22) - open/close 3117 (time 1760129154.22 total 30.03 last 119.76) - open/close 4408 (time 1760129164.22 total 40.03 last 129.06) total: 4500 open/close in 40.61 seconds: 110.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2806 (time 1760129176.77 total 10.00 last 280.57) total: 4500 /unlink in 15.98 seconds: 281.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1092 (time 1760129194.77 total 10.00 last 109.16) - open/close 2272 (time 1760129204.78 total 20.01 last 117.92) - open/close 3285 (time 1760129214.79 total 30.02 last 101.23) - open/close 4375 (time 1760129224.79 total 40.02 last 108.99) total: 4500 open/close in 41.20 seconds: 109.21 ops/second - /unlink 2341 (time 1760129238.59 total 10.00 last 234.07) - /unlink 4128 (time 1760129248.59 total 20.01 last 178.57) total: 4500 /unlink in 22.40 seconds: 200.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 972 (time 1760129263.87 total 10.00 last 97.19) - open/close 1900 (time 1760129273.87 total 20.00 last 92.79) - open/close 2804 (time 1760129283.88 total 30.01 last 90.33) - open/close 3640 (time 1760129293.88 total 40.01 last 83.58) - open/close 4461 (time 1760129303.88 total 50.02 last 82.06) total: 4500 open/close in 50.46 seconds: 89.18 ops/second - /unlink 1827 (time 1760129317.01 total 10.01 last 182.60) - /unlink 3264 (time 1760129327.02 total 20.01 last 143.64) total: 4500 /unlink in 27.61 seconds: 163.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 823 (time 1760129347.40 total 10.01 last 82.25) - open/close 1637 (time 1760129357.41 total 20.02 last 81.32) - open/close 2477 (time 1760129367.41 total 30.02 last 83.97) - open/close 3451 (time 1760129377.42 total 40.03 last 97.32) total: 4500 open/close in 49.18 seconds: 91.49 ops/second - /unlink 2041 (time 1760129398.90 total 10.00 last 204.00) - /unlink 3988 (time 1760129408.90 total 20.01 last 194.63) total: 4500 /unlink in 23.02 seconds: 195.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 1018 (time 1760129424.74 total 10.01 last 101.74) - open/close 2086 (time 1760129434.75 total 20.01 last 106.72) - open/close 3131 (time 1760129444.75 total 30.01 last 104.49) - open/close 4105 (time 1760129454.76 total 40.02 last 97.33) total: 4500 open/close in 43.55 seconds: 103.33 ops/second - /unlink 2220 (time 1760129470.59 total 10.00 last 221.91) - /unlink 4293 (time 1760129480.59 total 20.01 last 207.25) total: 4500 /unlink in 21.11 seconds: 213.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1303 (time 1760129494.00 total 10.01 last 130.18) - open/close 2388 (time 1760129504.00 total 20.01 last 108.47) - open/close 3492 (time 1760129514.01 total 30.02 last 110.31) total: 4500 open/close in 39.30 seconds: 114.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1994 (time 1760129535.59 total 10.01 last 199.29) - /unlink 4125 (time 1760129545.59 total 20.01 last 213.09) total: 4500 /unlink in 22.07 seconds: 203.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1422 (time 1760129559.87 total 10.01 last 142.10) - open/close 2654 (time 1760129569.88 total 20.01 last 123.12) - open/close 3706 (time 1760129579.88 total 30.02 last 105.17) total: 4500 open/close in 36.77 seconds: 122.40 ops/second - /unlink 2548 (time 1760129598.76 total 10.01 last 254.66) total: 4500 /unlink in 18.80 seconds: 239.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1217 (time 1760129619.61 total 10.01 last 121.57) - open/close 2490 (time 1760129629.62 total 20.01 last 127.28) - open/close 3575 (time 1760129639.63 total 30.03 last 108.34) total: 4500 open/close in 39.00 seconds: 115.38 ops/second - /unlink 2583 (time 1760129660.75 total 10.00 last 258.27) total: 4500 /unlink in 17.76 seconds: 253.38 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11154: 93725 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.