== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:53:12 (1781236392) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg335-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg335-server oleg335-server: oleg335-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg335-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 667 (time 1781236592.83 total 10.01 last 66.62) - open/close 1315 (time 1781236602.86 total 20.04 last 64.62) - open/close 1985 (time 1781236612.86 total 30.04 last 67.00) - open/close 2694 (time 1781236622.87 total 40.04 last 70.86) - open/close 3427 (time 1781236632.88 total 50.05 last 73.22) - open/close 4107 (time 1781236642.88 total 60.06 last 67.96) total: 4500 open/close in 66.51 seconds: 67.66 ops/second - /unlink 1186 (time 1781236663.49 total 10.00 last 118.55) - /unlink 2304 (time 1781236673.50 total 20.01 last 111.78) - /unlink 3427 (time 1781236683.50 total 30.01 last 112.24) total: 4500 /unlink in 39.26 seconds: 114.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 655 (time 1781236708.01 total 10.01 last 65.44) - open/close 1270 (time 1781236718.01 total 20.02 last 61.45) - open/close 1962 (time 1781236728.02 total 30.03 last 69.14) - open/close 2620 (time 1781236738.02 total 40.03 last 65.79) - open/close 3286 (time 1781236748.03 total 50.03 last 66.55) - open/close 3985 (time 1781236758.04 total 60.04 last 69.85) total: 4500 open/close in 69.93 seconds: 64.35 ops/second - /unlink 1137 (time 1781236783.53 total 10.00 last 113.66) - /unlink 2141 (time 1781236793.53 total 20.00 last 100.39) - /unlink 3274 (time 1781236803.54 total 30.01 last 113.20) - /unlink 4488 (time 1781236813.55 total 40.02 last 121.30) total: 4500 /unlink in 40.09 seconds: 112.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 586 (time 1781236828.58 total 10.02 last 58.51) - open/close 1195 (time 1781236838.59 total 20.03 last 60.79) - open/close 1855 (time 1781236848.59 total 30.03 last 66.00) - open/close 2548 (time 1781236858.61 total 40.05 last 69.18) - open/close 3166 (time 1781236868.61 total 50.05 last 61.77) - open/close 3869 (time 1781236878.62 total 60.06 last 70.26) - open/close 4416 (time 1781236888.64 total 70.08 last 54.59) total: 4500 open/close in 71.74 seconds: 62.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 908 (time 1781236905.93 total 10.00 last 90.76) - /unlink 1742 (time 1781236915.94 total 20.01 last 83.36) - /unlink 2800 (time 1781236925.94 total 30.01 last 105.80) - /unlink 3868 (time 1781236935.94 total 40.01 last 106.77) total: 4500 /unlink in 46.11 seconds: 97.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 614 (time 1781236958.17 total 10.01 last 61.37) - open/close 1213 (time 1781236968.19 total 20.02 last 59.79) - open/close 1902 (time 1781236978.20 total 30.03 last 68.85) - open/close 2517 (time 1781236988.20 total 40.03 last 61.49) - open/close 3242 (time 1781236998.20 total 50.04 last 72.47) - open/close 3904 (time 1781237008.21 total 60.05 last 66.12) total: 4500 open/close in 69.06 seconds: 65.16 ops/second - /unlink 1416 (time 1781237031.90 total 10.00 last 141.60) - /unlink 2702 (time 1781237041.91 total 20.01 last 128.50) - /unlink 3921 (time 1781237051.92 total 30.02 last 121.80) total: 4500 /unlink in 34.84 seconds: 129.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 696 (time 1781237072.09 total 10.01 last 69.56) - open/close 1390 (time 1781237082.10 total 20.02 last 69.32) - open/close 2104 (time 1781237092.11 total 30.02 last 71.37) - open/close 2674 (time 1781237102.11 total 40.03 last 56.97) - open/close 3367 (time 1781237112.11 total 50.03 last 69.29) - open/close 4040 (time 1781237122.12 total 60.04 last 67.22) total: 4500 open/close in 66.28 seconds: 67.89 ops/second - /unlink 1356 (time 1781237143.25 total 10.01 last 135.51) - /unlink 2720 (time 1781237153.25 total 20.01 last 136.40) - /unlink 3919 (time 1781237163.25 total 30.01 last 119.84) total: 4500 /unlink in 35.33 seconds: 127.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 716 (time 1781237182.44 total 10.02 last 71.48) - open/close 1427 (time 1781237192.45 total 20.03 last 71.03) - open/close 2076 (time 1781237202.46 total 30.04 last 64.83) - open/close 2816 (time 1781237212.46 total 40.04 last 73.96) - open/close 3556 (time 1781237222.47 total 50.05 last 73.94) - open/close 4349 (time 1781237232.47 total 60.05 last 79.29) total: 4500 open/close in 61.99 seconds: 72.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1337 (time 1781237248.64 total 10.01 last 133.58) - /unlink 2638 (time 1781237258.64 total 20.01 last 130.07) - /unlink 3918 (time 1781237268.64 total 30.02 last 127.94) total: 4500 /unlink in 34.63 seconds: 129.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 637 (time 1781237287.62 total 10.01 last 63.64) - open/close 1380 (time 1781237297.63 total 20.02 last 74.24) - open/close 2090 (time 1781237307.65 total 30.03 last 70.89) - open/close 2732 (time 1781237317.66 total 40.05 last 64.11) - open/close 3418 (time 1781237327.67 total 50.06 last 68.52) - open/close 4156 (time 1781237337.69 total 60.07 last 73.69) total: 4500 open/close in 65.70 seconds: 68.50 ops/second - /unlink 1250 (time 1781237357.40 total 10.01 last 124.89) - /unlink 2380 (time 1781237367.40 total 20.01 last 112.98) - /unlink 3501 (time 1781237377.40 total 30.01 last 112.09) total: 4500 /unlink in 38.86 seconds: 115.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 792 (time 1781237400.99 total 10.00 last 79.19) - open/close 1515 (time 1781237411.00 total 20.01 last 72.25) - open/close 2156 (time 1781237421.02 total 30.03 last 63.99) - open/close 2887 (time 1781237431.02 total 40.03 last 73.06) - open/close 3615 (time 1781237441.03 total 50.04 last 72.76) - open/close 4338 (time 1781237451.04 total 60.04 last 72.24) total: 4500 open/close in 62.79 seconds: 71.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1237 (time 1781237468.76 total 10.00 last 123.69) - /unlink 2456 (time 1781237478.77 total 20.01 last 121.84) - /unlink 3697 (time 1781237488.77 total 30.01 last 124.05) total: 4500 /unlink in 37.65 seconds: 119.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 626 (time 1781237512.39 total 10.01 last 62.52) - open/close 1241 (time 1781237522.40 total 20.02 last 61.47) - open/close 1858 (time 1781237532.40 total 30.03 last 61.66) - open/close 2558 (time 1781237542.41 total 40.04 last 69.92) - open/close 3266 (time 1781237552.42 total 50.04 last 70.76) - open/close 3868 (time 1781237562.45 total 60.07 last 60.04) - open/close 4428 (time 1781237572.47 total 70.09 last 55.88) total: 4500 open/close in 71.27 seconds: 63.14 ops/second - /unlink 1198 (time 1781237589.09 total 10.02 last 119.59) - /unlink 2391 (time 1781237599.10 total 20.02 last 119.24) - /unlink 3561 (time 1781237609.11 total 30.03 last 116.88) total: 4500 /unlink in 37.25 seconds: 120.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 722 (time 1781237631.35 total 10.00 last 72.19) - open/close 1497 (time 1781237641.35 total 20.00 last 77.49) - open/close 2153 (time 1781237651.35 total 30.00 last 65.59) - open/close 2868 (time 1781237661.37 total 40.02 last 71.42) - open/close 3590 (time 1781237671.38 total 50.02 last 72.13) - open/close 4317 (time 1781237681.39 total 60.04 last 72.62) total: 4500 open/close in 62.40 seconds: 72.11 ops/second - /unlink 1270 (time 1781237698.23 total 10.01 last 126.84) - /unlink 2436 (time 1781237708.23 total 20.01 last 116.58) - /unlink 3758 (time 1781237718.23 total 30.02 last 132.17) total: 4500 /unlink in 36.31 seconds: 123.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 681 (time 1781237739.48 total 10.03 last 67.87) - open/close 1408 (time 1781237749.48 total 20.03 last 72.70) - open/close 2106 (time 1781237759.49 total 30.04 last 69.73) - open/close 2796 (time 1781237769.50 total 40.05 last 68.96) - open/close 3434 (time 1781237779.50 total 50.05 last 63.77) - open/close 4119 (time 1781237789.51 total 60.06 last 68.45) total: 4500 open/close in 66.69 seconds: 67.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1254 (time 1781237810.35 total 10.00 last 125.36) - /unlink 2354 (time 1781237820.36 total 20.01 last 109.92) - /unlink 3609 (time 1781237830.36 total 30.01 last 125.45) total: 4500 /unlink in 37.84 seconds: 118.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 700 (time 1781237853.10 total 10.00 last 69.99) - open/close 1301 (time 1781237863.10 total 20.01 last 60.07) - open/close 1965 (time 1781237873.11 total 30.01 last 66.37) - open/close 2594 (time 1781237883.13 total 40.03 last 62.79) - open/close 3241 (time 1781237893.13 total 50.03 last 64.70) - open/close 3838 (time 1781237903.14 total 60.04 last 59.63) total: 4500 open/close in 69.67 seconds: 64.59 ops/second - /unlink 1181 (time 1781237927.19 total 10.00 last 118.08) - /unlink 2288 (time 1781237937.19 total 20.00 last 110.69) - /unlink 3583 (time 1781237947.20 total 30.01 last 129.40) total: 4500 /unlink in 37.80 seconds: 119.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 629 (time 1781237969.71 total 10.00 last 62.89) - open/close 1286 (time 1781237979.72 total 20.02 last 65.61) - open/close 1984 (time 1781237989.75 total 30.04 last 69.61) - open/close 2706 (time 1781237999.76 total 40.05 last 72.15) - open/close 3413 (time 1781238009.76 total 50.05 last 70.66) - open/close 4166 (time 1781238019.77 total 60.07 last 75.22) total: 4500 open/close in 64.54 seconds: 69.73 ops/second - /unlink 1242 (time 1781238038.88 total 10.00 last 124.16) - /unlink 2521 (time 1781238048.90 total 20.02 last 127.73) - /unlink 3668 (time 1781238058.90 total 30.02 last 114.65) total: 4500 /unlink in 37.23 seconds: 120.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 621 (time 1781238081.70 total 10.01 last 62.04) - open/close 1238 (time 1781238091.70 total 20.01 last 61.70) - open/close 1978 (time 1781238101.71 total 30.02 last 73.91) - open/close 2702 (time 1781238111.72 total 40.03 last 72.32) - open/close 3405 (time 1781238121.74 total 50.05 last 70.20) - open/close 4201 (time 1781238131.74 total 60.05 last 79.56) total: 4500 open/close in 64.43 seconds: 69.84 ops/second - /unlink 1309 (time 1781238149.95 total 10.01 last 130.73) - /unlink 2526 (time 1781238159.95 total 20.02 last 121.65) - /unlink 3826 (time 1781238169.96 total 30.02 last 129.92) total: 4500 /unlink in 35.27 seconds: 127.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 692 (time 1781238189.47 total 10.00 last 69.18) - open/close 1440 (time 1781238199.49 total 20.02 last 74.70) - open/close 2151 (time 1781238209.49 total 30.02 last 71.07) - open/close 2806 (time 1781238219.50 total 40.02 last 65.47) - open/close 3481 (time 1781238229.51 total 50.04 last 67.39) - open/close 4120 (time 1781238239.52 total 60.05 last 63.87) total: 4500 open/close in 66.52 seconds: 67.65 ops/second - /unlink 1191 (time 1781238261.62 total 10.00 last 119.08) - /unlink 2455 (time 1781238271.63 total 20.01 last 126.34) - /unlink 3751 (time 1781238281.63 total 30.01 last 129.52) total: 4500 /unlink in 36.60 seconds: 122.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 626 (time 1781238304.47 total 10.03 last 62.44) - open/close 1172 (time 1781238314.48 total 20.04 last 54.54) - open/close 1800 (time 1781238324.51 total 30.06 last 62.64) - open/close 2541 (time 1781238334.52 total 40.08 last 74.00) - open/close 3308 (time 1781238344.52 total 50.08 last 76.70) - open/close 4018 (time 1781238354.52 total 60.08 last 70.99) total: 4500 open/close in 67.02 seconds: 67.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1181 (time 1781238375.34 total 10.00 last 118.08) - /unlink 2426 (time 1781238385.35 total 20.01 last 124.37) - /unlink 3646 (time 1781238395.36 total 30.01 last 121.99) total: 4500 /unlink in 36.86 seconds: 122.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 660 (time 1781238416.73 total 10.02 last 65.85) - open/close 1390 (time 1781238426.75 total 20.04 last 72.87) - open/close 2138 (time 1781238436.75 total 30.05 last 74.75) - open/close 2903 (time 1781238446.76 total 40.06 last 76.44) - open/close 3633 (time 1781238456.77 total 50.07 last 72.92) - open/close 4350 (time 1781238466.78 total 60.07 last 71.67) total: 4500 open/close in 61.95 seconds: 72.64 ops/second - /unlink 1022 (time 1781238483.90 total 10.00 last 102.18) - /unlink 2288 (time 1781238493.91 total 20.01 last 126.53) - /unlink 3472 (time 1781238503.91 total 30.01 last 118.37) total: 4500 /unlink in 38.51 seconds: 116.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 740 (time 1781238527.42 total 10.00 last 73.99) - open/close 1496 (time 1781238537.43 total 20.01 last 75.54) - open/close 2204 (time 1781238547.43 total 30.01 last 70.79) - open/close 2853 (time 1781238557.44 total 40.02 last 64.84) - open/close 3530 (time 1781238567.44 total 50.02 last 67.66) - open/close 4128 (time 1781238577.45 total 60.03 last 59.78) total: 4500 open/close in 66.55 seconds: 67.62 ops/second - /unlink 1329 (time 1781238598.39 total 10.00 last 132.86) - /unlink 2504 (time 1781238608.40 total 20.01 last 117.42) - /unlink 3838 (time 1781238618.40 total 30.01 last 133.34) total: 4500 /unlink in 34.94 seconds: 128.80 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94120 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: modules unloaded.