== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:24:53 (1788459893) Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg421-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg421-server oleg421-server: oleg421-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: kmemleak disabled oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg421-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg421-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 673 (time 1788460118.82 total 10.00 last 67.30) - open/close 1294 (time 1788460128.83 total 20.01 last 62.07) - open/close 1964 (time 1788460138.84 total 30.01 last 66.94) - open/close 2611 (time 1788460148.85 total 40.03 last 64.63) - open/close 3223 (time 1788460158.87 total 50.05 last 61.05) - open/close 3792 (time 1788460168.88 total 60.06 last 56.86) - open/close 4414 (time 1788460178.89 total 70.07 last 62.15) total: 4500 open/close in 71.46 seconds: 62.98 ops/second - /unlink 1292 (time 1788460195.95 total 10.00 last 129.20) - /unlink 2539 (time 1788460205.95 total 20.00 last 124.70) - /unlink 3811 (time 1788460215.96 total 30.01 last 127.13) total: 4500 /unlink in 36.03 seconds: 124.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 511 (time 1788460237.65 total 10.00 last 51.08) - open/close 1169 (time 1788460247.65 total 20.01 last 65.76) - open/close 1784 (time 1788460257.66 total 30.02 last 61.45) - open/close 2361 (time 1788460267.67 total 40.03 last 57.65) - open/close 2972 (time 1788460277.67 total 50.03 last 61.09) - open/close 3643 (time 1788460287.68 total 60.04 last 67.04) - open/close 4293 (time 1788460297.69 total 70.05 last 64.95) total: 4500 open/close in 73.41 seconds: 61.30 ops/second - /unlink 1252 (time 1788460315.76 total 10.00 last 125.19) - /unlink 2463 (time 1788460325.76 total 20.00 last 121.05) - /unlink 3676 (time 1788460335.77 total 30.01 last 121.24) total: 4500 /unlink in 36.66 seconds: 122.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 476 (time 1788460358.52 total 10.01 last 47.55) - open/close 1064 (time 1788460368.53 total 20.03 last 58.71) - open/close 1786 (time 1788460378.54 total 30.03 last 72.18) - open/close 2441 (time 1788460388.55 total 40.04 last 65.40) - open/close 3157 (time 1788460398.56 total 50.05 last 71.55) - open/close 3805 (time 1788460408.56 total 60.06 last 64.78) - open/close 4476 (time 1788460418.57 total 70.06 last 67.04) total: 4500 open/close in 70.34 seconds: 63.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1248 (time 1788460433.54 total 10.00 last 124.75) - /unlink 2489 (time 1788460443.56 total 20.02 last 123.89) - /unlink 3684 (time 1788460453.56 total 30.02 last 119.46) total: 4500 /unlink in 36.77 seconds: 122.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 600 (time 1788460475.89 total 10.01 last 59.96) - open/close 1296 (time 1788460485.89 total 20.01 last 69.59) - open/close 2002 (time 1788460495.89 total 30.01 last 70.57) - open/close 2664 (time 1788460505.90 total 40.02 last 66.12) - open/close 3330 (time 1788460515.91 total 50.03 last 66.56) - open/close 3992 (time 1788460525.91 total 60.03 last 66.16) total: 4500 open/close in 67.12 seconds: 67.04 ops/second - /unlink 1296 (time 1788460547.93 total 10.01 last 129.42) - /unlink 2557 (time 1788460557.93 total 20.01 last 126.09) - /unlink 3894 (time 1788460567.93 total 30.02 last 133.67) total: 4500 /unlink in 34.58 seconds: 130.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 740 (time 1788460587.88 total 10.02 last 73.87) - open/close 1428 (time 1788460597.88 total 20.02 last 68.78) - open/close 2100 (time 1788460607.89 total 30.03 last 67.15) - open/close 2870 (time 1788460617.90 total 40.04 last 76.93) - open/close 3532 (time 1788460627.90 total 50.04 last 66.20) - open/close 4136 (time 1788460637.90 total 60.04 last 60.36) total: 4500 open/close in 65.60 seconds: 68.59 ops/second - /unlink 1155 (time 1788460659.19 total 10.01 last 115.38) - /unlink 2198 (time 1788460669.20 total 20.01 last 104.25) - /unlink 3444 (time 1788460679.20 total 30.01 last 124.60) total: 4500 /unlink in 38.08 seconds: 118.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 577 (time 1788460702.92 total 10.03 last 57.55) - open/close 1252 (time 1788460712.93 total 20.03 last 67.45) - open/close 1868 (time 1788460722.94 total 30.04 last 61.57) - open/close 2472 (time 1788460732.94 total 40.04 last 60.38) - open/close 3241 (time 1788460742.95 total 50.05 last 76.83) - open/close 3891 (time 1788460752.95 total 60.06 last 64.96) total: 4500 open/close in 69.83 seconds: 64.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1185 (time 1788460778.26 total 10.01 last 118.43) - /unlink 2521 (time 1788460788.26 total 20.01 last 133.55) - /unlink 3774 (time 1788460798.26 total 30.01 last 125.24) total: 4500 /unlink in 35.31 seconds: 127.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 675 (time 1788460818.68 total 10.01 last 67.44) - open/close 1421 (time 1788460828.69 total 20.02 last 74.54) - open/close 2041 (time 1788460838.70 total 30.02 last 61.96) - open/close 2740 (time 1788460848.70 total 40.03 last 69.88) - open/close 3415 (time 1788460858.71 total 50.03 last 67.45) - open/close 4147 (time 1788460868.72 total 60.04 last 73.12) total: 4500 open/close in 65.87 seconds: 68.32 ops/second - /unlink 1307 (time 1788460889.67 total 10.00 last 130.66) - /unlink 2420 (time 1788460899.68 total 20.00 last 111.28) - /unlink 3460 (time 1788460909.68 total 30.01 last 103.93) total: 4500 /unlink in 37.94 seconds: 118.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 649 (time 1788460932.15 total 10.03 last 64.73) - open/close 1258 (time 1788460942.16 total 20.04 last 60.81) - open/close 1837 (time 1788460952.17 total 30.05 last 57.86) - open/close 2484 (time 1788460962.19 total 40.06 last 64.60) - open/close 3089 (time 1788460972.19 total 50.07 last 60.46) - open/close 3807 (time 1788460982.20 total 60.07 last 71.78) total: 4500 open/close in 69.43 seconds: 64.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1237 (time 1788461007.32 total 10.00 last 123.66) - /unlink 2474 (time 1788461017.32 total 20.00 last 123.69) - /unlink 3831 (time 1788461027.32 total 30.01 last 135.68) total: 4500 /unlink in 34.99 seconds: 128.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 626 (time 1788461048.09 total 10.01 last 62.55) - open/close 1304 (time 1788461058.09 total 20.02 last 67.75) - open/close 1815 (time 1788461068.10 total 30.02 last 51.06) - open/close 2448 (time 1788461078.11 total 40.03 last 63.23) - open/close 3128 (time 1788461088.11 total 50.03 last 68.00) - open/close 3857 (time 1788461098.12 total 60.04 last 72.87) total: 4500 open/close in 68.51 seconds: 65.69 ops/second - /unlink 1315 (time 1788461121.70 total 10.01 last 131.43) - /unlink 2522 (time 1788461131.70 total 20.01 last 120.69) - /unlink 3724 (time 1788461141.71 total 30.01 last 120.16) total: 4500 /unlink in 35.02 seconds: 128.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 630 (time 1788461162.41 total 10.01 last 62.93) - open/close 1234 (time 1788461172.42 total 20.02 last 60.33) - open/close 1906 (time 1788461182.43 total 30.03 last 67.17) - open/close 2631 (time 1788461192.44 total 40.03 last 72.44) - open/close 3304 (time 1788461202.44 total 50.03 last 67.30) - open/close 3917 (time 1788461212.48 total 60.08 last 61.04) total: 4500 open/close in 69.72 seconds: 64.54 ops/second - /unlink 1070 (time 1788461237.33 total 10.01 last 106.93) - /unlink 2108 (time 1788461247.33 total 20.01 last 103.72) - /unlink 3548 (time 1788461257.34 total 30.02 last 143.93) total: 4500 /unlink in 38.05 seconds: 118.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 632 (time 1788461281.53 total 10.00 last 63.17) - open/close 1243 (time 1788461291.54 total 20.01 last 61.04) - open/close 1842 (time 1788461301.54 total 30.02 last 59.89) - open/close 2535 (time 1788461311.54 total 40.02 last 69.29) - open/close 3169 (time 1788461321.55 total 50.02 last 63.38) - open/close 3841 (time 1788461331.55 total 60.02 last 67.20) - open/close 4495 (time 1788461341.56 total 70.03 last 65.33) total: 4500 open/close in 70.14 seconds: 64.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1030 (time 1788461356.37 total 10.02 last 102.83) - /unlink 2182 (time 1788461366.38 total 20.02 last 115.13) - /unlink 3323 (time 1788461376.38 total 30.02 last 114.09) - /unlink 4398 (time 1788461386.38 total 40.02 last 107.49) total: 4500 /unlink in 41.12 seconds: 109.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 550 (time 1788461403.77 total 10.00 last 54.98) - open/close 1114 (time 1788461413.77 total 20.01 last 56.36) - open/close 1805 (time 1788461423.78 total 30.01 last 69.08) - open/close 2500 (time 1788461433.81 total 40.05 last 69.25) - open/close 3160 (time 1788461443.82 total 50.06 last 65.94) - open/close 3878 (time 1788461453.83 total 60.07 last 71.72) total: 4500 open/close in 69.69 seconds: 64.58 ops/second - /unlink 1201 (time 1788461478.26 total 10.00 last 120.08) - /unlink 2512 (time 1788461488.27 total 20.01 last 130.97) - /unlink 3747 (time 1788461498.28 total 30.02 last 123.45) total: 4500 /unlink in 35.64 seconds: 126.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 590 (time 1788461518.88 total 10.00 last 58.98) - open/close 1171 (time 1788461528.89 total 20.01 last 58.04) - open/close 1710 (time 1788461538.91 total 30.03 last 53.79) - open/close 2289 (time 1788461548.92 total 40.04 last 57.85) - open/close 2920 (time 1788461558.95 total 50.07 last 62.95) - open/close 3602 (time 1788461568.96 total 60.08 last 68.12) - open/close 4233 (time 1788461578.96 total 70.08 last 63.07) total: 4500 open/close in 74.20 seconds: 60.65 ops/second - /unlink 1203 (time 1788461598.54 total 10.00 last 120.30) - /unlink 2357 (time 1788461608.54 total 20.01 last 115.34) - /unlink 3661 (time 1788461618.54 total 30.01 last 130.39) total: 4500 /unlink in 36.19 seconds: 124.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 711 (time 1788461639.37 total 10.00 last 71.07) - open/close 1389 (time 1788461649.37 total 20.01 last 67.77) - open/close 2109 (time 1788461659.38 total 30.01 last 71.97) - open/close 2730 (time 1788461669.40 total 40.03 last 61.98) - open/close 3374 (time 1788461679.41 total 50.04 last 64.33) - open/close 3903 (time 1788461689.43 total 60.06 last 52.79) - open/close 4491 (time 1788461699.43 total 70.07 last 58.79) total: 4500 open/close in 70.15 seconds: 64.15 ops/second - /unlink 1070 (time 1788461714.47 total 10.00 last 106.96) - /unlink 2188 (time 1788461724.49 total 20.02 last 111.65) - /unlink 3395 (time 1788461734.49 total 30.02 last 120.66) - /unlink 4395 (time 1788461744.49 total 40.02 last 99.99) total: 4500 /unlink in 40.94 seconds: 109.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 731 (time 1788461761.64 total 10.00 last 73.09) - open/close 1417 (time 1788461771.64 total 20.00 last 68.59) - open/close 2084 (time 1788461781.66 total 30.02 last 66.58) - open/close 2808 (time 1788461791.68 total 40.04 last 72.26) - open/close 3466 (time 1788461801.68 total 50.04 last 65.78) - open/close 4142 (time 1788461811.69 total 60.05 last 67.57) total: 4500 open/close in 65.95 seconds: 68.24 ops/second - /unlink 1093 (time 1788461831.95 total 10.01 last 109.23) - /unlink 2336 (time 1788461841.95 total 20.01 last 124.25) - /unlink 3522 (time 1788461851.96 total 30.01 last 118.59) total: 4500 /unlink in 38.93 seconds: 115.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 643 (time 1788461876.47 total 10.03 last 64.10) - open/close 1285 (time 1788461886.47 total 20.03 last 64.18) - open/close 1956 (time 1788461896.48 total 30.04 last 67.07) - open/close 2643 (time 1788461906.49 total 40.05 last 68.63) - open/close 3204 (time 1788461916.50 total 50.06 last 56.03) - open/close 3907 (time 1788461926.51 total 60.07 last 70.23) total: 4500 open/close in 68.91 seconds: 65.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1272 (time 1788461949.65 total 10.00 last 127.16) - /unlink 2545 (time 1788461959.65 total 20.01 last 127.25) - /unlink 3743 (time 1788461969.65 total 30.01 last 119.80) total: 4500 /unlink in 36.97 seconds: 121.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 642 (time 1788461991.74 total 10.02 last 64.10) - open/close 1237 (time 1788462001.74 total 20.02 last 59.47) - open/close 1808 (time 1788462011.75 total 30.03 last 57.07) - open/close 2464 (time 1788462021.76 total 40.04 last 65.54) - open/close 3082 (time 1788462031.77 total 50.05 last 61.72) - open/close 3732 (time 1788462041.78 total 60.06 last 64.92) - open/close 4312 (time 1788462051.78 total 70.06 last 58.00) total: 4500 open/close in 73.88 seconds: 60.91 ops/second - /unlink 1148 (time 1788462070.19 total 10.01 last 114.70) - /unlink 2361 (time 1788462080.20 total 20.02 last 121.16) - /unlink 3302 (time 1788462090.21 total 30.03 last 94.00) total: 4500 /unlink in 39.75 seconds: 113.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 591 (time 1788462116.14 total 10.01 last 59.05) - open/close 1247 (time 1788462126.16 total 20.03 last 65.48) - open/close 1848 (time 1788462136.16 total 30.03 last 60.07) - open/close 2425 (time 1788462146.17 total 40.04 last 57.64) - open/close 3066 (time 1788462156.18 total 50.05 last 64.06) - open/close 3681 (time 1788462166.19 total 60.06 last 61.42) - open/close 4287 (time 1788462176.21 total 70.08 last 60.48) total: 4500 open/close in 73.11 seconds: 61.55 ops/second - /unlink 1385 (time 1788462193.50 total 10.00 last 138.50) - /unlink 2636 (time 1788462203.50 total 20.01 last 124.98) - /unlink 3906 (time 1788462213.51 total 30.01 last 126.99) total: 4500 /unlink in 35.06 seconds: 128.37 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95364 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: kmemleak disabled modules unloaded.