== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:02:15 (1788444135) Stopping clients: oleg614-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg614-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg614-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg614-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg614-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg614-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg614-server oleg614-server: oleg614-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg614-server' oleg614-server: oleg614-server.virtnet: executing load_modules_local oleg614-server: Loading modules from /home/green/git/lustre-release/lustre oleg614-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg614-server: detected 4 online CPUs by sysfs oleg614-server: MODOPTS_LIBCFS= oleg614-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg614-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg614-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg614-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg614-client.virtnet: -o user_xattr,flock 192.168.206.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 823 (time 1788444332.10 total 10.01 last 82.20) - open/close 1558 (time 1788444342.11 total 20.02 last 73.44) - open/close 2369 (time 1788444352.13 total 30.04 last 80.96) - open/close 2985 (time 1788444362.14 total 40.05 last 61.51) - open/close 3725 (time 1788444372.14 total 50.05 last 74.00) - open/close 4452 (time 1788444382.14 total 60.05 last 72.69) total: 4500 open/close in 60.76 seconds: 74.06 ops/second - /unlink 1299 (time 1788444397.44 total 10.01 last 129.83) - /unlink 2733 (time 1788444407.44 total 20.01 last 143.37) - /unlink 4327 (time 1788444417.44 total 30.01 last 159.40) total: 4500 /unlink in 31.93 seconds: 140.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 845 (time 1788444435.28 total 10.00 last 84.46) - open/close 1651 (time 1788444445.29 total 20.01 last 80.58) - open/close 2537 (time 1788444455.29 total 30.01 last 88.58) - open/close 3404 (time 1788444465.30 total 40.02 last 86.59) - open/close 4086 (time 1788444475.30 total 50.02 last 68.18) total: 4500 open/close in 56.62 seconds: 79.47 ops/second - /unlink 1681 (time 1788444495.81 total 10.00 last 168.04) - /unlink 3334 (time 1788444505.82 total 20.01 last 165.25) total: 4500 /unlink in 26.65 seconds: 168.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 934 (time 1788444526.97 total 10.01 last 93.35) - open/close 1887 (time 1788444536.97 total 20.01 last 95.27) - open/close 2734 (time 1788444546.98 total 30.02 last 84.64) - open/close 3600 (time 1788444556.99 total 40.02 last 86.56) total: 4500 open/close in 49.63 seconds: 90.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1366 (time 1788444581.16 total 10.04 last 136.01) - /unlink 2403 (time 1788444591.17 total 20.06 last 103.55) - /unlink 3456 (time 1788444601.17 total 30.06 last 105.30) total: 4500 /unlink in 37.51 seconds: 119.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 817 (time 1788444624.13 total 10.00 last 81.70) - open/close 1583 (time 1788444634.14 total 20.00 last 76.59) - open/close 2410 (time 1788444644.14 total 30.00 last 82.69) - open/close 3299 (time 1788444654.14 total 40.01 last 88.87) - open/close 4139 (time 1788444664.15 total 50.01 last 83.94) total: 4500 open/close in 54.11 seconds: 83.16 ops/second - /unlink 1723 (time 1788444682.33 total 10.00 last 172.28) - /unlink 3204 (time 1788444692.33 total 20.00 last 148.06) total: 4500 /unlink in 28.81 seconds: 156.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 826 (time 1788444715.86 total 10.01 last 82.48) - open/close 1657 (time 1788444725.87 total 20.02 last 83.07) - open/close 2486 (time 1788444735.87 total 30.02 last 82.86) - open/close 3327 (time 1788444745.91 total 40.06 last 83.82) - open/close 4203 (time 1788444755.92 total 50.07 last 87.49) total: 4500 open/close in 53.43 seconds: 84.22 ops/second - /unlink 1536 (time 1788444772.97 total 10.01 last 153.42) - /unlink 3216 (time 1788444782.97 total 20.01 last 167.99) total: 4500 /unlink in 29.46 seconds: 152.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 792 (time 1788444806.86 total 10.01 last 79.10) - open/close 1536 (time 1788444816.87 total 20.02 last 74.35) - open/close 2439 (time 1788444826.87 total 30.02 last 90.30) - open/close 3323 (time 1788444836.87 total 40.02 last 88.39) - open/close 4245 (time 1788444846.88 total 50.03 last 92.13) total: 4500 open/close in 53.10 seconds: 84.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1576 (time 1788444863.39 total 10.00 last 157.54) - /unlink 3162 (time 1788444873.39 total 20.00 last 158.60) - /unlink 4482 (time 1788444883.40 total 30.01 last 131.97) total: 4500 /unlink in 30.07 seconds: 149.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 835 (time 1788444897.63 total 10.02 last 83.30) - open/close 1656 (time 1788444907.68 total 20.07 last 81.73) - open/close 2617 (time 1788444917.68 total 30.07 last 96.07) - open/close 3451 (time 1788444927.69 total 40.08 last 83.33) - open/close 4285 (time 1788444937.69 total 50.09 last 83.34) total: 4500 open/close in 52.57 seconds: 85.61 ops/second - /unlink 1596 (time 1788444954.44 total 10.00 last 159.53) - /unlink 3246 (time 1788444964.44 total 20.00 last 165.00) total: 4500 /unlink in 27.72 seconds: 162.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 877 (time 1788444985.54 total 10.00 last 87.68) - open/close 1757 (time 1788444995.55 total 20.01 last 87.91) - open/close 2529 (time 1788445005.55 total 30.01 last 77.20) - open/close 3209 (time 1788445015.55 total 40.02 last 67.98) - open/close 3744 (time 1788445025.56 total 50.02 last 53.46) - open/close 4447 (time 1788445035.56 total 60.03 last 70.28) total: 4500 open/close in 60.92 seconds: 73.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1522 (time 1788445051.00 total 10.01 last 152.08) - /unlink 3033 (time 1788445061.00 total 20.01 last 151.08) total: 4500 /unlink in 29.38 seconds: 153.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 795 (time 1788445084.74 total 10.01 last 79.44) - open/close 1569 (time 1788445094.76 total 20.03 last 77.25) - open/close 2363 (time 1788445104.76 total 30.03 last 79.36) - open/close 3191 (time 1788445114.78 total 40.05 last 82.68) - open/close 3919 (time 1788445124.79 total 50.06 last 72.69) total: 4500 open/close in 58.64 seconds: 76.74 ops/second - /unlink 1329 (time 1788445147.73 total 10.01 last 132.75) - /unlink 2845 (time 1788445157.74 total 20.01 last 151.59) - /unlink 4389 (time 1788445167.74 total 30.02 last 154.31) total: 4500 /unlink in 31.10 seconds: 144.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 723 (time 1788445184.27 total 10.01 last 72.24) - open/close 1521 (time 1788445194.27 total 20.01 last 79.78) - open/close 2402 (time 1788445204.28 total 30.02 last 88.03) - open/close 3259 (time 1788445214.29 total 40.02 last 85.66) - open/close 4008 (time 1788445224.29 total 50.03 last 74.88) total: 4500 open/close in 58.47 seconds: 76.97 ops/second - /unlink 1032 (time 1788445248.64 total 10.01 last 103.14) - /unlink 2087 (time 1788445258.64 total 20.01 last 105.48) - /unlink 3615 (time 1788445268.64 total 30.01 last 152.79) total: 4500 /unlink in 36.18 seconds: 124.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 915 (time 1788445288.65 total 10.00 last 91.49) - open/close 1802 (time 1788445298.65 total 20.01 last 88.65) - open/close 2693 (time 1788445308.66 total 30.01 last 89.05) - open/close 3566 (time 1788445318.67 total 40.02 last 87.20) - open/close 4436 (time 1788445328.67 total 50.03 last 86.95) total: 4500 open/close in 50.73 seconds: 88.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1798 (time 1788445342.51 total 10.00 last 179.76) - /unlink 3414 (time 1788445352.52 total 20.01 last 161.48) total: 4500 /unlink in 26.48 seconds: 169.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 653 (time 1788445372.91 total 10.00 last 65.30) - open/close 1492 (time 1788445382.92 total 20.01 last 83.80) - open/close 2292 (time 1788445392.95 total 30.04 last 79.82) - open/close 3129 (time 1788445402.95 total 40.04 last 83.68) - open/close 3954 (time 1788445412.96 total 50.05 last 82.42) total: 4500 open/close in 58.45 seconds: 76.99 ops/second - /unlink 1147 (time 1788445436.61 total 10.01 last 114.58) - /unlink 2537 (time 1788445446.61 total 20.01 last 138.97) - /unlink 3860 (time 1788445456.61 total 30.01 last 132.29) total: 4500 /unlink in 34.53 seconds: 130.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 871 (time 1788445475.77 total 10.00 last 87.06) - open/close 1715 (time 1788445485.79 total 20.02 last 84.29) - open/close 2523 (time 1788445495.79 total 30.02 last 80.79) - open/close 3373 (time 1788445505.79 total 40.02 last 84.99) - open/close 4207 (time 1788445515.80 total 50.03 last 83.28) total: 4500 open/close in 53.16 seconds: 84.65 ops/second - /unlink 1369 (time 1788445533.00 total 10.00 last 136.85) - /unlink 2819 (time 1788445543.01 total 20.01 last 144.96) - /unlink 4278 (time 1788445553.01 total 30.01 last 145.88) total: 4500 /unlink in 31.65 seconds: 142.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 878 (time 1788445569.39 total 10.02 last 87.62) - open/close 1668 (time 1788445579.40 total 20.03 last 78.92) - open/close 2534 (time 1788445589.40 total 30.04 last 86.55) - open/close 3387 (time 1788445599.41 total 40.05 last 85.21) - open/close 4174 (time 1788445609.41 total 50.05 last 78.70) total: 4500 open/close in 54.19 seconds: 83.04 ops/second - /unlink 1541 (time 1788445628.38 total 10.00 last 154.06) - /unlink 2773 (time 1788445638.38 total 20.01 last 123.16) - /unlink 3994 (time 1788445648.38 total 30.01 last 122.06) total: 4500 /unlink in 35.79 seconds: 125.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 823 (time 1788445669.13 total 10.02 last 82.15) - open/close 1648 (time 1788445679.13 total 20.02 last 82.48) - open/close 2558 (time 1788445689.13 total 30.02 last 90.98) - open/close 3388 (time 1788445699.16 total 40.05 last 82.78) - open/close 4293 (time 1788445709.17 total 50.06 last 90.43) total: 4500 open/close in 52.68 seconds: 85.43 ops/second - /unlink 1499 (time 1788445727.15 total 10.01 last 149.70) - /unlink 3217 (time 1788445737.16 total 20.02 last 171.75) total: 4500 /unlink in 28.68 seconds: 156.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 952 (time 1788445760.51 total 10.00 last 95.18) - open/close 1780 (time 1788445770.52 total 20.02 last 82.69) - open/close 2612 (time 1788445780.53 total 30.02 last 83.19) - open/close 3435 (time 1788445790.53 total 40.02 last 82.27) - open/close 4188 (time 1788445800.53 total 50.03 last 75.27) total: 4500 open/close in 54.03 seconds: 83.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1687 (time 1788445819.55 total 10.00 last 168.66) - /unlink 3336 (time 1788445829.55 total 20.00 last 164.87) total: 4500 /unlink in 26.96 seconds: 166.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 780 (time 1788445850.47 total 10.01 last 77.94) - open/close 1575 (time 1788445860.49 total 20.03 last 79.34) - open/close 2400 (time 1788445870.49 total 30.03 last 82.49) - open/close 3299 (time 1788445880.49 total 40.03 last 89.87) - open/close 4165 (time 1788445890.49 total 50.03 last 86.59) total: 4500 open/close in 53.52 seconds: 84.08 ops/second - /unlink 1647 (time 1788445908.93 total 10.00 last 164.69) - /unlink 3186 (time 1788445918.93 total 20.00 last 153.86) total: 4500 /unlink in 29.62 seconds: 151.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 772 (time 1788445943.50 total 10.01 last 77.14) - open/close 1593 (time 1788445953.50 total 20.01 last 82.06) - open/close 2322 (time 1788445963.50 total 30.02 last 72.88) - open/close 3047 (time 1788445973.51 total 40.02 last 72.45) - open/close 3810 (time 1788445983.51 total 50.02 last 76.30) total: 4500 open/close in 58.63 seconds: 76.75 ops/second - /unlink 1459 (time 1788446006.35 total 10.01 last 145.78) - /unlink 3013 (time 1788446016.36 total 20.02 last 155.23) total: 4500 /unlink in 29.96 seconds: 150.20 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95198 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg614-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg614-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg614-server stop mds service on oleg614-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg614-server stop mds service on oleg614-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg614-server unloading modules via unload_modules_local on: 'oleg614-server' oleg614-server: oleg614-server.virtnet: executing unload_modules_local oleg614-server: modules unloaded.