== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:25:49 (1788445549) Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg315-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg315-server oleg315-server: oleg315-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg315-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg315-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 637 (time 1788445771.50 total 10.04 last 63.47) - open/close 1283 (time 1788445781.51 total 20.06 last 64.48) - open/close 1856 (time 1788445791.52 total 30.06 last 57.25) - open/close 2454 (time 1788445801.54 total 40.08 last 59.72) - open/close 3062 (time 1788445811.54 total 50.08 last 60.78) - open/close 3718 (time 1788445821.55 total 60.09 last 65.54) - open/close 4330 (time 1788445831.55 total 70.09 last 61.17) total: 4500 open/close in 73.10 seconds: 61.56 ops/second - /unlink 1250 (time 1788445849.39 total 10.00 last 124.95) - /unlink 2401 (time 1788445859.40 total 20.01 last 115.03) - /unlink 3710 (time 1788445869.41 total 30.02 last 130.79) total: 4500 /unlink in 36.31 seconds: 123.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 690 (time 1788445891.14 total 10.00 last 68.99) - open/close 1287 (time 1788445901.15 total 20.01 last 59.66) - open/close 1912 (time 1788445911.17 total 30.03 last 62.39) - open/close 2460 (time 1788445921.17 total 40.03 last 54.78) - open/close 3094 (time 1788445931.21 total 50.07 last 63.15) - open/close 3664 (time 1788445941.21 total 60.07 last 57.00) - open/close 4257 (time 1788445951.21 total 70.07 last 59.29) total: 4500 open/close in 73.36 seconds: 61.34 ops/second - /unlink 1334 (time 1788445969.24 total 10.01 last 133.25) - /unlink 2480 (time 1788445979.24 total 20.01 last 114.57) - /unlink 3666 (time 1788445989.24 total 30.02 last 118.58) total: 4500 /unlink in 36.75 seconds: 122.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 549 (time 1788446011.47 total 10.01 last 54.87) - open/close 1059 (time 1788446021.48 total 20.01 last 50.96) - open/close 1714 (time 1788446031.49 total 30.02 last 65.42) - open/close 2207 (time 1788446041.51 total 40.04 last 49.24) - open/close 2813 (time 1788446051.51 total 50.04 last 60.60) - open/close 3389 (time 1788446061.52 total 60.05 last 57.51) - open/close 4004 (time 1788446071.53 total 70.06 last 61.46) total: 4500 open/close in 78.92 seconds: 57.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1076 (time 1788446097.22 total 10.01 last 107.52) - /unlink 2111 (time 1788446107.23 total 20.01 last 103.43) - /unlink 3296 (time 1788446117.23 total 30.02 last 118.44) total: 4500 /unlink in 39.48 seconds: 113.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 688 (time 1788446141.44 total 10.02 last 68.64) - open/close 1319 (time 1788446151.45 total 20.03 last 63.05) - open/close 1884 (time 1788446161.45 total 30.03 last 56.49) - open/close 2571 (time 1788446171.47 total 40.05 last 68.58) - open/close 3191 (time 1788446181.47 total 50.06 last 61.97) - open/close 3851 (time 1788446191.48 total 60.06 last 65.95) total: 4500 open/close in 69.07 seconds: 65.16 ops/second - /unlink 1197 (time 1788446215.62 total 10.01 last 119.53) - /unlink 2302 (time 1788446225.62 total 20.02 last 110.49) - /unlink 3423 (time 1788446235.63 total 30.02 last 112.10) total: 4500 /unlink in 38.51 seconds: 116.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 662 (time 1788446260.21 total 10.02 last 66.08) - open/close 1260 (time 1788446270.22 total 20.03 last 59.74) - open/close 1974 (time 1788446280.24 total 30.05 last 71.25) - open/close 2620 (time 1788446290.24 total 40.06 last 64.56) - open/close 3282 (time 1788446300.25 total 50.06 last 66.15) - open/close 3980 (time 1788446310.26 total 60.07 last 69.72) total: 4500 open/close in 67.16 seconds: 67.00 ops/second - /unlink 1301 (time 1788446331.78 total 10.01 last 130.00) - /unlink 2573 (time 1788446341.78 total 20.01 last 127.18) - /unlink 3829 (time 1788446351.79 total 30.01 last 125.55) total: 4500 /unlink in 35.19 seconds: 127.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 675 (time 1788446371.04 total 10.03 last 67.33) - open/close 1317 (time 1788446381.06 total 20.04 last 64.10) - open/close 1819 (time 1788446391.10 total 30.08 last 50.00) - open/close 2531 (time 1788446401.11 total 40.09 last 71.16) - open/close 3254 (time 1788446411.12 total 50.10 last 72.23) - open/close 3933 (time 1788446421.13 total 60.11 last 67.79) total: 4500 open/close in 67.95 seconds: 66.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1196 (time 1788446443.66 total 10.00 last 119.56) - /unlink 2535 (time 1788446453.66 total 20.01 last 133.86) - /unlink 3779 (time 1788446463.67 total 30.01 last 124.33) total: 4500 /unlink in 35.75 seconds: 125.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 681 (time 1788446484.12 total 10.01 last 68.02) - open/close 1389 (time 1788446494.13 total 20.02 last 70.75) - open/close 2022 (time 1788446504.14 total 30.03 last 63.26) - open/close 2718 (time 1788446514.14 total 40.03 last 69.57) - open/close 3429 (time 1788446524.15 total 50.04 last 71.01) - open/close 4151 (time 1788446534.16 total 60.04 last 72.19) total: 4500 open/close in 64.79 seconds: 69.45 ops/second - /unlink 1229 (time 1788446553.39 total 10.00 last 122.84) - /unlink 2527 (time 1788446563.39 total 20.01 last 129.78) - /unlink 3923 (time 1788446573.39 total 30.01 last 139.59) total: 4500 /unlink in 34.37 seconds: 130.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 705 (time 1788446591.97 total 10.00 last 70.49) - open/close 1350 (time 1788446601.99 total 20.02 last 64.40) - open/close 2036 (time 1788446612.00 total 30.02 last 68.56) - open/close 2694 (time 1788446622.00 total 40.02 last 65.80) - open/close 3359 (time 1788446632.00 total 50.02 last 66.50) - open/close 3974 (time 1788446642.00 total 60.02 last 61.49) total: 4500 open/close in 68.45 seconds: 65.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1279 (time 1788446664.27 total 10.00 last 127.88) - /unlink 2554 (time 1788446674.27 total 20.00 last 127.48) - /unlink 3796 (time 1788446684.27 total 30.01 last 124.16) total: 4500 /unlink in 36.60 seconds: 122.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 663 (time 1788446705.51 total 10.02 last 66.19) - open/close 1365 (time 1788446715.52 total 20.03 last 70.14) - open/close 2110 (time 1788446725.52 total 30.03 last 74.45) - open/close 2784 (time 1788446735.52 total 40.04 last 67.39) - open/close 3492 (time 1788446745.54 total 50.05 last 70.71) - open/close 4209 (time 1788446755.54 total 60.05 last 71.69) total: 4500 open/close in 64.25 seconds: 70.04 ops/second - /unlink 1356 (time 1788446773.67 total 10.00 last 135.58) - /unlink 2625 (time 1788446783.67 total 20.01 last 126.82) - /unlink 3981 (time 1788446793.68 total 30.02 last 135.50) total: 4500 /unlink in 33.93 seconds: 132.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 723 (time 1788446812.01 total 10.01 last 72.21) - open/close 1446 (time 1788446822.03 total 20.03 last 72.21) - open/close 2126 (time 1788446832.04 total 30.04 last 67.91) - open/close 2826 (time 1788446842.05 total 40.05 last 69.95) - open/close 3497 (time 1788446852.05 total 50.05 last 67.09) - open/close 4216 (time 1788446862.05 total 60.05 last 71.87) total: 4500 open/close in 64.32 seconds: 69.96 ops/second - /unlink 1222 (time 1788446880.64 total 10.00 last 122.20) - /unlink 2282 (time 1788446890.65 total 20.00 last 105.99) - /unlink 3565 (time 1788446900.65 total 30.00 last 128.29) total: 4500 /unlink in 37.16 seconds: 121.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 703 (time 1788446922.65 total 10.01 last 70.20) - open/close 1408 (time 1788446932.65 total 20.02 last 70.47) - open/close 2126 (time 1788446942.66 total 30.02 last 71.77) - open/close 2826 (time 1788446952.66 total 40.03 last 69.97) - open/close 3497 (time 1788446962.68 total 50.05 last 66.97) - open/close 4251 (time 1788446972.69 total 60.06 last 75.33) total: 4500 open/close in 63.60 seconds: 70.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1332 (time 1788446990.45 total 10.00 last 133.15) - /unlink 2495 (time 1788447000.45 total 20.00 last 116.30) - /unlink 3842 (time 1788447010.45 total 30.01 last 134.67) total: 4500 /unlink in 34.99 seconds: 128.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 718 (time 1788447029.70 total 10.00 last 71.78) - open/close 1431 (time 1788447039.71 total 20.02 last 71.21) - open/close 2179 (time 1788447049.72 total 30.03 last 74.73) - open/close 2894 (time 1788447059.73 total 40.03 last 71.46) - open/close 3575 (time 1788447069.75 total 50.06 last 67.94) - open/close 4228 (time 1788447079.76 total 60.06 last 65.26) total: 4500 open/close in 63.69 seconds: 70.66 ops/second - /unlink 1366 (time 1788447097.42 total 10.02 last 136.39) - /unlink 2614 (time 1788447107.42 total 20.02 last 124.77) - /unlink 3934 (time 1788447117.43 total 30.02 last 131.94) total: 4500 /unlink in 34.23 seconds: 131.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 688 (time 1788447136.03 total 10.01 last 68.72) - open/close 1392 (time 1788447146.06 total 20.04 last 70.22) - open/close 2025 (time 1788447156.07 total 30.05 last 63.23) - open/close 2708 (time 1788447166.10 total 40.08 last 68.08) - open/close 3429 (time 1788447176.11 total 50.09 last 72.07) - open/close 4101 (time 1788447186.11 total 60.09 last 67.15) total: 4500 open/close in 65.76 seconds: 68.44 ops/second - /unlink 1114 (time 1788447206.19 total 10.02 last 111.16) - /unlink 2242 (time 1788447216.19 total 20.02 last 112.80) - /unlink 3346 (time 1788447226.20 total 30.03 last 110.33) total: 4500 /unlink in 39.11 seconds: 115.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 735 (time 1788447250.79 total 10.03 last 73.29) - open/close 1472 (time 1788447260.79 total 20.03 last 73.67) - open/close 2150 (time 1788447270.80 total 30.04 last 67.75) - open/close 2851 (time 1788447280.81 total 40.05 last 70.01) - open/close 3544 (time 1788447290.82 total 50.06 last 69.26) - open/close 4225 (time 1788447300.82 total 60.06 last 68.07) total: 4500 open/close in 63.92 seconds: 70.40 ops/second - /unlink 1283 (time 1788447319.12 total 10.01 last 128.15) - /unlink 2493 (time 1788447329.12 total 20.01 last 121.00) - /unlink 3739 (time 1788447339.12 total 30.02 last 124.56) total: 4500 /unlink in 35.62 seconds: 126.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 477 (time 1788447360.18 total 10.00 last 47.70) - open/close 1030 (time 1788447370.18 total 20.01 last 55.26) - open/close 1723 (time 1788447380.19 total 30.01 last 69.28) - open/close 2458 (time 1788447390.19 total 40.01 last 73.47) - open/close 3234 (time 1788447400.20 total 50.02 last 77.55) - open/close 3897 (time 1788447410.20 total 60.02 last 66.27) total: 4500 open/close in 68.78 seconds: 65.42 ops/second - /unlink 1300 (time 1788447433.21 total 10.00 last 129.97) - /unlink 2493 (time 1788447443.21 total 20.01 last 119.23) - /unlink 3569 (time 1788447453.23 total 30.02 last 107.46) total: 4500 /unlink in 38.75 seconds: 116.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 670 (time 1788447477.11 total 10.01 last 66.96) - open/close 1321 (time 1788447487.11 total 20.01 last 65.10) - open/close 1989 (time 1788447497.11 total 30.01 last 66.79) - open/close 2707 (time 1788447507.11 total 40.01 last 71.75) - open/close 3335 (time 1788447517.12 total 50.02 last 62.76) - open/close 3979 (time 1788447527.14 total 60.04 last 64.30) total: 4500 open/close in 68.44 seconds: 65.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1190 (time 1788447550.48 total 10.01 last 118.92) - /unlink 2461 (time 1788447560.48 total 20.01 last 127.09) - /unlink 3602 (time 1788447570.49 total 30.02 last 114.00) total: 4500 /unlink in 37.58 seconds: 119.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 673 (time 1788447593.27 total 10.01 last 67.25) - open/close 1281 (time 1788447603.28 total 20.02 last 60.74) - open/close 1880 (time 1788447613.29 total 30.03 last 59.83) - open/close 2621 (time 1788447623.29 total 40.03 last 74.10) - open/close 3294 (time 1788447633.30 total 50.04 last 67.25) - open/close 3964 (time 1788447643.31 total 60.05 last 66.89) total: 4500 open/close in 69.00 seconds: 65.22 ops/second - /unlink 1278 (time 1788447666.75 total 10.00 last 127.74) - /unlink 2443 (time 1788447676.76 total 20.01 last 116.45) - /unlink 3667 (time 1788447686.76 total 30.01 last 122.35) total: 4500 /unlink in 36.89 seconds: 121.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 640 (time 1788447708.56 total 10.00 last 63.98) - open/close 1306 (time 1788447718.56 total 20.01 last 66.55) - open/close 1925 (time 1788447728.56 total 30.01 last 61.90) - open/close 2561 (time 1788447738.57 total 40.01 last 63.58) - open/close 3288 (time 1788447748.57 total 50.02 last 72.64) - open/close 3977 (time 1788447758.60 total 60.05 last 68.71) total: 4500 open/close in 66.86 seconds: 67.31 ops/second - /unlink 1236 (time 1788447779.98 total 10.01 last 123.53) - /unlink 2412 (time 1788447789.98 total 20.01 last 117.56) - /unlink 3621 (time 1788447799.99 total 30.01 last 120.86) total: 4500 /unlink in 36.62 seconds: 122.87 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95359 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local oleg315-server: modules unloaded.