== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:59:43 (1789127983) Stopping clients: oleg446-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg446-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg446-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg446-server oleg446-server: oleg446-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing load_modules_local oleg446-server: Loading modules from /home/green/git/lustre-release/lustre oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg446-server: detected 4 online CPUs by sysfs oleg446-server: MODOPTS_LIBCFS= oleg446-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg446-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg446-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg446-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 683 (time 1789128208.91 total 10.01 last 68.23) - open/close 1417 (time 1789128218.92 total 20.02 last 73.36) - open/close 2108 (time 1789128228.93 total 30.03 last 68.98) - open/close 2779 (time 1789128238.94 total 40.04 last 67.07) - open/close 3463 (time 1789128248.95 total 50.05 last 68.34) - open/close 4154 (time 1789128258.95 total 60.05 last 69.05) total: 4500 open/close in 64.75 seconds: 69.50 ops/second - /unlink 1112 (time 1789128278.49 total 10.00 last 111.15) - /unlink 2277 (time 1789128288.53 total 20.04 last 116.06) - /unlink 3399 (time 1789128298.53 total 30.05 last 112.16) total: 4500 /unlink in 38.88 seconds: 115.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 616 (time 1789128322.19 total 10.01 last 61.55) - open/close 1289 (time 1789128332.20 total 20.03 last 67.17) - open/close 1944 (time 1789128342.22 total 30.05 last 65.38) - open/close 2558 (time 1789128352.22 total 40.05 last 61.40) - open/close 3183 (time 1789128362.23 total 50.05 last 62.46) - open/close 3883 (time 1789128372.23 total 60.05 last 69.98) total: 4500 open/close in 69.18 seconds: 65.04 ops/second - /unlink 1205 (time 1789128395.94 total 10.00 last 120.50) - /unlink 2374 (time 1789128405.95 total 20.01 last 116.79) - /unlink 3585 (time 1789128415.95 total 30.01 last 121.08) total: 4500 /unlink in 37.36 seconds: 120.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 630 (time 1789128438.09 total 10.00 last 62.98) - open/close 1250 (time 1789128448.09 total 20.01 last 61.97) - open/close 1914 (time 1789128458.11 total 30.03 last 66.28) - open/close 2561 (time 1789128468.11 total 40.03 last 64.68) - open/close 3256 (time 1789128478.13 total 50.05 last 69.35) - open/close 3859 (time 1789128488.14 total 60.06 last 60.27) total: 4500 open/close in 69.75 seconds: 64.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1279 (time 1789128512.85 total 10.00 last 127.89) - /unlink 2599 (time 1789128522.85 total 20.00 last 131.99) - /unlink 3863 (time 1789128532.86 total 30.01 last 126.34) total: 4500 /unlink in 34.54 seconds: 130.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 656 (time 1789128552.36 total 10.01 last 65.53) - open/close 1317 (time 1789128562.38 total 20.03 last 65.98) - open/close 1976 (time 1789128572.38 total 30.03 last 65.86) - open/close 2625 (time 1789128582.39 total 40.04 last 64.85) - open/close 3294 (time 1789128592.40 total 50.05 last 66.87) - open/close 4034 (time 1789128602.41 total 60.05 last 73.94) total: 4500 open/close in 66.73 seconds: 67.43 ops/second - /unlink 1236 (time 1789128623.09 total 10.00 last 123.60) - /unlink 2399 (time 1789128633.10 total 20.01 last 116.20) - /unlink 3610 (time 1789128643.10 total 30.01 last 121.08) total: 4500 /unlink in 36.51 seconds: 123.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 676 (time 1789128664.13 total 10.01 last 67.51) - open/close 1247 (time 1789128674.16 total 20.04 last 56.97) - open/close 1918 (time 1789128684.16 total 30.04 last 67.07) - open/close 2531 (time 1789128694.16 total 40.05 last 61.27) - open/close 3219 (time 1789128704.17 total 50.05 last 68.75) - open/close 3841 (time 1789128714.18 total 60.06 last 62.17) total: 4500 open/close in 69.41 seconds: 64.83 ops/second - /unlink 1300 (time 1789128737.48 total 10.00 last 130.00) - /unlink 2641 (time 1789128747.48 total 20.00 last 134.06) - /unlink 3990 (time 1789128757.49 total 30.01 last 134.75) total: 4500 /unlink in 34.05 seconds: 132.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 602 (time 1789128776.16 total 10.01 last 60.17) - open/close 1208 (time 1789128786.17 total 20.01 last 60.56) - open/close 1883 (time 1789128796.19 total 30.03 last 67.36) - open/close 2581 (time 1789128806.19 total 40.04 last 69.77) - open/close 3298 (time 1789128816.20 total 50.04 last 71.66) - open/close 3934 (time 1789128826.20 total 60.05 last 63.59) total: 4500 open/close in 69.92 seconds: 64.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 934 (time 1789128852.23 total 10.01 last 93.29) - /unlink 2053 (time 1789128862.23 total 20.02 last 111.86) - /unlink 3191 (time 1789128872.24 total 30.02 last 113.78) - /unlink 4448 (time 1789128882.24 total 40.02 last 125.66) total: 4500 /unlink in 40.47 seconds: 111.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 680 (time 1789128896.84 total 10.02 last 67.89) - open/close 1341 (time 1789128906.85 total 20.02 last 66.07) - open/close 2001 (time 1789128916.85 total 30.03 last 65.94) - open/close 2686 (time 1789128926.86 total 40.04 last 68.44) - open/close 3385 (time 1789128936.89 total 50.06 last 69.72) - open/close 4102 (time 1789128946.90 total 60.08 last 71.59) total: 4500 open/close in 66.29 seconds: 67.88 ops/second - /unlink 1246 (time 1789128968.00 total 10.00 last 124.57) - /unlink 2500 (time 1789128978.00 total 20.01 last 125.36) - /unlink 3775 (time 1789128988.00 total 30.01 last 127.45) total: 4500 /unlink in 36.20 seconds: 124.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 676 (time 1789129009.10 total 10.00 last 67.58) - open/close 1380 (time 1789129019.10 total 20.01 last 70.37) - open/close 2000 (time 1789129029.10 total 30.01 last 61.99) - open/close 2666 (time 1789129039.11 total 40.01 last 66.57) - open/close 3328 (time 1789129049.12 total 50.02 last 66.15) - open/close 3940 (time 1789129059.12 total 60.02 last 61.18) total: 4500 open/close in 68.31 seconds: 65.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1057 (time 1789129082.93 total 10.00 last 105.65) - /unlink 2351 (time 1789129092.94 total 20.01 last 129.30) - /unlink 3585 (time 1789129102.94 total 30.02 last 123.34) total: 4500 /unlink in 37.46 seconds: 120.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 662 (time 1789129125.23 total 10.00 last 66.18) - open/close 1285 (time 1789129135.24 total 20.01 last 62.24) - open/close 1891 (time 1789129145.25 total 30.02 last 60.59) - open/close 2462 (time 1789129155.25 total 40.02 last 57.07) - open/close 3100 (time 1789129165.27 total 50.04 last 63.70) - open/close 3664 (time 1789129175.27 total 60.04 last 56.36) - open/close 4278 (time 1789129185.29 total 70.06 last 61.31) total: 4500 open/close in 74.01 seconds: 60.81 ops/second - /unlink 1291 (time 1789129204.30 total 10.00 last 129.04) - /unlink 2515 (time 1789129214.31 total 20.02 last 122.24) - /unlink 3697 (time 1789129224.32 total 30.03 last 118.05) total: 4500 /unlink in 37.06 seconds: 121.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 662 (time 1789129247.48 total 10.02 last 66.05) - open/close 1310 (time 1789129257.49 total 20.03 last 64.78) - open/close 1960 (time 1789129267.49 total 30.03 last 64.98) - open/close 2586 (time 1789129277.49 total 40.03 last 62.60) - open/close 3280 (time 1789129287.50 total 50.04 last 69.34) - open/close 3899 (time 1789129297.51 total 60.05 last 61.82) total: 4500 open/close in 69.58 seconds: 64.68 ops/second - /unlink 1271 (time 1789129321.15 total 10.01 last 127.01) - /unlink 2536 (time 1789129331.16 total 20.02 last 126.34) - /unlink 3834 (time 1789129341.17 total 30.02 last 129.74) total: 4500 /unlink in 36.41 seconds: 123.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 656 (time 1789129362.19 total 10.00 last 65.58) - open/close 1203 (time 1789129372.21 total 20.02 last 54.58) - open/close 1822 (time 1789129382.25 total 30.06 last 61.66) - open/close 2515 (time 1789129392.25 total 40.06 last 69.29) - open/close 3187 (time 1789129402.27 total 50.08 last 67.10) - open/close 3826 (time 1789129412.27 total 60.09 last 63.84) total: 4500 open/close in 70.05 seconds: 64.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1084 (time 1789129436.77 total 10.00 last 108.40) - /unlink 2058 (time 1789129446.77 total 20.00 last 97.38) - /unlink 3059 (time 1789129456.77 total 30.01 last 100.07) - /unlink 4132 (time 1789129466.78 total 40.01 last 107.28) total: 4500 /unlink in 43.80 seconds: 102.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 598 (time 1789129485.99 total 10.00 last 59.77) - open/close 1159 (time 1789129496.00 total 20.01 last 56.08) - open/close 1738 (time 1789129506.02 total 30.03 last 57.79) - open/close 2387 (time 1789129516.03 total 40.04 last 64.79) - open/close 3008 (time 1789129526.04 total 50.05 last 62.08) - open/close 3553 (time 1789129536.04 total 60.05 last 54.48) - open/close 4174 (time 1789129546.05 total 70.06 last 62.06) total: 4500 open/close in 75.49 seconds: 59.61 ops/second - /unlink 1260 (time 1789129565.92 total 10.00 last 125.95) - /unlink 2432 (time 1789129575.93 total 20.01 last 117.17) - /unlink 3555 (time 1789129585.94 total 30.02 last 112.18) total: 4500 /unlink in 38.29 seconds: 117.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 622 (time 1789129610.56 total 10.01 last 62.15) - open/close 1238 (time 1789129620.57 total 20.02 last 61.56) - open/close 1836 (time 1789129630.59 total 30.03 last 59.69) - open/close 2435 (time 1789129640.59 total 40.04 last 59.88) - open/close 3126 (time 1789129650.60 total 50.04 last 69.05) - open/close 3785 (time 1789129660.61 total 60.05 last 65.84) - open/close 4478 (time 1789129670.63 total 70.07 last 69.17) total: 4500 open/close in 70.32 seconds: 64.00 ops/second - /unlink 1251 (time 1789129685.42 total 10.03 last 124.76) - /unlink 2340 (time 1789129695.44 total 20.05 last 108.68) - /unlink 3205 (time 1789129705.44 total 30.05 last 86.47) - /unlink 4130 (time 1789129715.45 total 40.06 last 92.47) total: 4500 /unlink in 43.08 seconds: 104.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 480 (time 1789129734.34 total 10.01 last 47.94) - open/close 1016 (time 1789129744.35 total 20.03 last 53.51) - open/close 1617 (time 1789129754.36 total 30.04 last 60.07) - open/close 2220 (time 1789129764.36 total 40.04 last 60.29) - open/close 2741 (time 1789129774.37 total 50.05 last 52.04) - open/close 3309 (time 1789129784.38 total 60.06 last 56.75) - open/close 3809 (time 1789129794.39 total 70.07 last 49.96) - open/close 4284 (time 1789129804.40 total 80.07 last 47.46) total: 4500 open/close in 83.95 seconds: 53.60 ops/second - /unlink 739 (time 1789129825.33 total 10.01 last 73.85) - /unlink 1628 (time 1789129835.34 total 20.02 last 88.82) - /unlink 2672 (time 1789129845.34 total 30.02 last 104.39) - /unlink 3843 (time 1789129855.35 total 40.02 last 117.02) total: 4500 /unlink in 46.84 seconds: 96.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 535 (time 1789129879.96 total 10.00 last 53.49) - open/close 1112 (time 1789129889.98 total 20.02 last 57.61) - open/close 1642 (time 1789129899.98 total 30.02 last 52.98) - open/close 2158 (time 1789129909.99 total 40.03 last 51.56) - open/close 2748 (time 1789129919.99 total 50.03 last 59.00) - open/close 3411 (time 1789129930.01 total 60.05 last 66.19) - open/close 4063 (time 1789129940.01 total 70.05 last 65.19) total: 4500 open/close in 77.31 seconds: 58.21 ops/second - /unlink 1162 (time 1789129962.10 total 10.00 last 116.20) - /unlink 2281 (time 1789129972.10 total 20.00 last 111.88) - /unlink 3413 (time 1789129982.11 total 30.01 last 113.07) - /unlink 4430 (time 1789129992.11 total 40.02 last 101.68) total: 4500 /unlink in 40.69 seconds: 110.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 602 (time 1789130008.61 total 10.02 last 60.11) - open/close 1128 (time 1789130018.62 total 20.02 last 52.56) - open/close 1686 (time 1789130028.63 total 30.03 last 55.75) - open/close 2283 (time 1789130038.64 total 40.04 last 59.67) - open/close 2889 (time 1789130048.65 total 50.05 last 60.49) - open/close 3404 (time 1789130058.67 total 60.07 last 51.42) - open/close 3957 (time 1789130068.68 total 70.08 last 55.24) - open/close 4468 (time 1789130078.69 total 80.09 last 51.04) total: 4500 open/close in 80.79 seconds: 55.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1185 (time 1789130094.77 total 10.00 last 118.44) - /unlink 2212 (time 1789130104.77 total 20.01 last 102.67) - /unlink 3155 (time 1789130114.78 total 30.01 last 94.27) - /unlink 4131 (time 1789130124.78 total 40.01 last 97.58) total: 4500 /unlink in 44.03 seconds: 102.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 505 (time 1789130144.92 total 10.01 last 50.46) - open/close 1072 (time 1789130154.96 total 20.04 last 56.52) - open/close 1614 (time 1789130164.97 total 30.06 last 54.11) - open/close 2143 (time 1789130174.98 total 40.07 last 52.84) - open/close 2697 (time 1789130184.98 total 50.07 last 55.40) - open/close 3261 (time 1789130194.99 total 60.08 last 56.36) - open/close 3759 (time 1789130204.99 total 70.08 last 49.79) - open/close 4372 (time 1789130215.01 total 80.09 last 61.24) total: 4500 open/close in 82.14 seconds: 54.78 ops/second - /unlink 1180 (time 1789130232.70 total 10.00 last 117.99) - /unlink 2101 (time 1789130242.71 total 20.01 last 92.00) - /unlink 2979 (time 1789130252.72 total 30.02 last 87.75) - /unlink 3801 (time 1789130262.73 total 40.03 last 82.13) total: 4500 /unlink in 47.57 seconds: 94.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 592 (time 1789130285.25 total 10.01 last 59.15) - open/close 1181 (time 1789130295.29 total 20.05 last 58.64) - open/close 1785 (time 1789130305.31 total 30.08 last 60.25) - open/close 2352 (time 1789130315.32 total 40.08 last 56.67) - open/close 2999 (time 1789130325.33 total 50.09 last 64.65) - open/close 3605 (time 1789130335.34 total 60.10 last 60.56) - open/close 4201 (time 1789130345.38 total 70.14 last 59.34) total: 4500 open/close in 74.69 seconds: 60.25 ops/second - /unlink 1197 (time 1789130365.00 total 10.01 last 119.53) - /unlink 2333 (time 1789130375.00 total 20.02 last 113.57) - /unlink 3397 (time 1789130385.01 total 30.03 last 106.30) total: 4500 /unlink in 38.70 seconds: 116.29 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95326 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg446-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server stop mds service on oleg446-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server stop mds service on oleg446-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server unloading modules via unload_modules_local on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing unload_modules_local oleg446-server: modules unloaded.