== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:21:10 (1787851270) Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg134-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg134-server oleg134-server: oleg134-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg134-server: detected 4 online CPUs by sysfs oleg134-server: MODOPTS_LIBCFS= oleg134-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg134-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg134-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg134-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 595 (time 1787851496.40 total 10.03 last 59.35) - open/close 1303 (time 1787851506.45 total 20.07 last 70.48) - open/close 1907 (time 1787851516.46 total 30.08 last 60.34) - open/close 2423 (time 1787851526.46 total 40.08 last 51.59) - open/close 2969 (time 1787851536.49 total 50.11 last 54.43) - open/close 3638 (time 1787851546.50 total 60.12 last 66.83) - open/close 4365 (time 1787851556.50 total 70.13 last 72.68) total: 4500 open/close in 72.75 seconds: 61.86 ops/second - /unlink 1302 (time 1787851574.63 total 10.00 last 130.14) - /unlink 2534 (time 1787851584.67 total 20.04 last 122.71) - /unlink 3731 (time 1787851594.68 total 30.05 last 119.58) total: 4500 /unlink in 36.59 seconds: 122.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 596 (time 1787851617.77 total 10.00 last 59.59) - open/close 1174 (time 1787851627.79 total 20.02 last 57.71) - open/close 2041 (time 1787851637.80 total 30.03 last 86.63) - open/close 2760 (time 1787851647.80 total 40.03 last 71.89) - open/close 3532 (time 1787851657.80 total 50.03 last 77.14) - open/close 4246 (time 1787851667.81 total 60.04 last 71.33) total: 4500 open/close in 64.13 seconds: 70.17 ops/second - /unlink 1093 (time 1787851686.27 total 10.00 last 109.28) - /unlink 1864 (time 1787851696.28 total 20.01 last 77.05) - /unlink 3181 (time 1787851706.28 total 30.01 last 131.68) - /unlink 4324 (time 1787851716.30 total 40.03 last 114.11) total: 4500 /unlink in 42.18 seconds: 106.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 771 (time 1787851734.07 total 10.01 last 77.00) - open/close 1264 (time 1787851744.08 total 20.03 last 49.22) - open/close 1877 (time 1787851754.11 total 30.06 last 61.11) - open/close 2419 (time 1787851764.13 total 40.08 last 54.10) - open/close 3067 (time 1787851774.15 total 50.09 last 64.70) - open/close 3650 (time 1787851784.15 total 60.10 last 58.28) - open/close 4332 (time 1787851794.15 total 70.10 last 68.17) total: 4500 open/close in 73.15 seconds: 61.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1043 (time 1787851815.28 total 10.00 last 104.27) - /unlink 1852 (time 1787851825.29 total 20.01 last 80.88) - /unlink 2905 (time 1787851835.29 total 30.01 last 105.28) - /unlink 4079 (time 1787851845.31 total 40.02 last 117.21) total: 4500 /unlink in 43.15 seconds: 104.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 689 (time 1787851864.43 total 10.00 last 68.87) - open/close 1343 (time 1787851874.44 total 20.01 last 65.38) - open/close 2015 (time 1787851884.45 total 30.02 last 67.10) - open/close 2777 (time 1787851894.46 total 40.03 last 76.16) - open/close 3497 (time 1787851904.46 total 50.03 last 71.99) - open/close 4210 (time 1787851914.46 total 60.03 last 71.28) total: 4500 open/close in 64.57 seconds: 69.69 ops/second - /unlink 1397 (time 1787851934.16 total 10.00 last 139.65) - /unlink 2883 (time 1787851944.16 total 20.01 last 148.53) - /unlink 4258 (time 1787851954.17 total 30.01 last 137.47) total: 4500 /unlink in 31.91 seconds: 141.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 713 (time 1787851971.26 total 10.01 last 71.22) - open/close 1288 (time 1787851981.28 total 20.03 last 57.41) - open/close 1887 (time 1787851991.28 total 30.03 last 59.90) - open/close 2335 (time 1787852001.30 total 40.05 last 44.70) - open/close 2947 (time 1787852011.33 total 50.07 last 61.05) - open/close 3568 (time 1787852021.33 total 60.08 last 62.05) - open/close 4308 (time 1787852031.34 total 70.09 last 73.95) total: 4500 open/close in 72.57 seconds: 62.01 ops/second - /unlink 1337 (time 1787852048.60 total 10.00 last 133.67) - /unlink 2727 (time 1787852058.61 total 20.01 last 138.92) - /unlink 4072 (time 1787852068.61 total 30.01 last 134.50) total: 4500 /unlink in 32.51 seconds: 138.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 665 (time 1787852087.34 total 10.01 last 66.42) - open/close 1375 (time 1787852097.34 total 20.02 last 70.97) - open/close 2087 (time 1787852107.35 total 30.02 last 71.15) - open/close 2675 (time 1787852117.35 total 40.03 last 58.77) - open/close 3350 (time 1787852127.36 total 50.03 last 67.47) - open/close 3943 (time 1787852137.39 total 60.06 last 59.13) total: 4500 open/close in 68.88 seconds: 65.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1338 (time 1787852161.72 total 10.00 last 133.80) - /unlink 2723 (time 1787852171.73 total 20.01 last 138.38) - /unlink 4068 (time 1787852181.75 total 30.03 last 134.27) total: 4500 /unlink in 33.96 seconds: 132.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 621 (time 1787852203.69 total 10.00 last 62.09) - open/close 1185 (time 1787852213.69 total 20.00 last 56.40) - open/close 1647 (time 1787852223.70 total 30.01 last 46.18) - open/close 2072 (time 1787852233.71 total 40.02 last 42.44) - open/close 2579 (time 1787852243.72 total 50.03 last 50.66) - open/close 3068 (time 1787852253.72 total 60.03 last 48.88) - open/close 3656 (time 1787852263.72 total 70.03 last 58.80) - open/close 4416 (time 1787852273.72 total 80.04 last 75.98) total: 4500 open/close in 80.96 seconds: 55.58 ops/second - /unlink 1410 (time 1787852289.49 total 10.00 last 140.94) - /unlink 2674 (time 1787852299.49 total 20.00 last 126.39) - /unlink 3894 (time 1787852309.50 total 30.01 last 121.94) total: 4500 /unlink in 34.83 seconds: 129.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 689 (time 1787852331.03 total 10.00 last 68.89) - open/close 1354 (time 1787852341.04 total 20.01 last 66.45) - open/close 2016 (time 1787852351.05 total 30.02 last 66.15) - open/close 2608 (time 1787852361.06 total 40.03 last 59.15) - open/close 3034 (time 1787852371.07 total 50.03 last 42.56) - open/close 3652 (time 1787852381.08 total 60.05 last 61.72) - open/close 4223 (time 1787852391.09 total 70.05 last 57.07) total: 4500 open/close in 73.65 seconds: 61.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1502 (time 1787852410.08 total 10.02 last 149.95) - /unlink 2882 (time 1787852420.09 total 20.02 last 137.92) - /unlink 4242 (time 1787852430.10 total 30.03 last 135.89) total: 4500 /unlink in 32.04 seconds: 140.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 747 (time 1787852447.38 total 10.02 last 74.57) - open/close 1514 (time 1787852457.39 total 20.03 last 76.63) - open/close 2193 (time 1787852467.41 total 30.04 last 67.78) - open/close 2774 (time 1787852477.43 total 40.07 last 57.97) - open/close 3290 (time 1787852487.44 total 50.07 last 51.56) - open/close 4046 (time 1787852497.44 total 60.07 last 75.59) total: 4500 open/close in 66.32 seconds: 67.85 ops/second - /unlink 1298 (time 1787852518.37 total 10.00 last 129.76) - /unlink 2610 (time 1787852528.38 total 20.01 last 131.07) - /unlink 3796 (time 1787852538.40 total 30.03 last 118.36) total: 4500 /unlink in 34.42 seconds: 130.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 409 (time 1787852560.13 total 10.07 last 40.61) - open/close 981 (time 1787852570.13 total 20.07 last 57.19) - open/close 1797 (time 1787852580.13 total 30.07 last 81.60) - open/close 2483 (time 1787852590.13 total 40.07 last 68.60) - open/close 3144 (time 1787852600.14 total 50.09 last 66.02) - open/close 3609 (time 1787852610.17 total 60.11 last 46.40) - open/close 4057 (time 1787852620.19 total 70.13 last 44.70) total: 4500 open/close in 78.60 seconds: 57.25 ops/second - /unlink 1456 (time 1787852643.66 total 10.01 last 145.48) - /unlink 2773 (time 1787852653.66 total 20.01 last 131.70) - /unlink 4149 (time 1787852663.66 total 30.01 last 137.60) total: 4500 /unlink in 34.63 seconds: 129.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 475 (time 1787852687.21 total 10.01 last 47.46) - open/close 1117 (time 1787852697.22 total 20.01 last 64.18) - open/close 1687 (time 1787852707.23 total 30.02 last 56.94) - open/close 2305 (time 1787852717.24 total 40.04 last 61.71) - open/close 2844 (time 1787852727.27 total 50.06 last 53.75) - open/close 3427 (time 1787852737.27 total 60.07 last 58.28) - open/close 4111 (time 1787852747.27 total 70.07 last 68.40) total: 4500 open/close in 75.00 seconds: 60.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1190 (time 1787852766.08 total 10.00 last 119.00) - /unlink 2403 (time 1787852776.08 total 20.00 last 121.25) - /unlink 3672 (time 1787852786.10 total 30.02 last 126.71) total: 4500 /unlink in 38.39 seconds: 117.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 664 (time 1787852810.62 total 10.01 last 66.35) - open/close 1382 (time 1787852820.63 total 20.02 last 71.73) - open/close 2140 (time 1787852830.67 total 30.06 last 75.48) - open/close 2701 (time 1787852840.67 total 40.06 last 56.08) - open/close 3241 (time 1787852850.69 total 50.08 last 53.89) - open/close 4076 (time 1787852860.69 total 60.08 last 83.49) total: 4500 open/close in 65.41 seconds: 68.80 ops/second - /unlink 1350 (time 1787852880.95 total 10.00 last 134.93) - /unlink 2848 (time 1787852890.95 total 20.01 last 149.77) - /unlink 4224 (time 1787852900.96 total 30.01 last 137.50) total: 4500 /unlink in 32.49 seconds: 138.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 444 (time 1787852919.34 total 10.00 last 44.39) - open/close 1048 (time 1787852929.35 total 20.00 last 60.39) - open/close 1775 (time 1787852939.35 total 30.01 last 72.68) - open/close 2526 (time 1787852949.35 total 40.01 last 75.08) - open/close 3255 (time 1787852959.36 total 50.02 last 72.84) - open/close 3944 (time 1787852969.38 total 60.04 last 68.74) total: 4500 open/close in 67.58 seconds: 66.59 ops/second - /unlink 1380 (time 1787852991.60 total 10.01 last 137.89) - /unlink 2846 (time 1787853001.60 total 20.01 last 146.51) - /unlink 4251 (time 1787853011.61 total 30.02 last 140.46) total: 4500 /unlink in 32.28 seconds: 139.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 703 (time 1787853029.43 total 10.01 last 70.26) - open/close 1326 (time 1787853039.44 total 20.01 last 62.26) - open/close 2000 (time 1787853049.45 total 30.02 last 67.33) - open/close 2706 (time 1787853059.46 total 40.03 last 70.54) - open/close 3414 (time 1787853069.46 total 50.03 last 70.78) - open/close 4152 (time 1787853079.46 total 60.04 last 73.77) total: 4500 open/close in 64.98 seconds: 69.25 ops/second - /unlink 1776 (time 1787853098.99 total 10.00 last 177.58) - /unlink 3032 (time 1787853108.99 total 20.00 last 125.59) - /unlink 4237 (time 1787853118.99 total 30.01 last 120.41) total: 4500 /unlink in 32.06 seconds: 140.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 688 (time 1787853136.81 total 10.01 last 68.72) - open/close 1355 (time 1787853146.81 total 20.01 last 66.70) - open/close 1912 (time 1787853156.82 total 30.03 last 55.61) - open/close 2542 (time 1787853166.84 total 40.05 last 62.88) - open/close 3105 (time 1787853176.85 total 50.06 last 56.24) - open/close 3748 (time 1787853186.87 total 60.07 last 64.22) - open/close 4426 (time 1787853196.87 total 70.08 last 67.75) total: 4500 open/close in 71.11 seconds: 63.28 ops/second - /unlink 1407 (time 1787853212.59 total 10.00 last 140.69) - /unlink 2714 (time 1787853222.59 total 20.00 last 130.68) - /unlink 4169 (time 1787853232.59 total 30.00 last 145.49) total: 4500 /unlink in 32.50 seconds: 138.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 723 (time 1787853251.18 total 10.00 last 72.29) - open/close 1368 (time 1787853261.19 total 20.01 last 64.47) - open/close 2107 (time 1787853271.19 total 30.01 last 73.84) - open/close 2775 (time 1787853281.20 total 40.02 last 66.77) - open/close 3398 (time 1787853291.20 total 50.02 last 62.28) - open/close 4019 (time 1787853301.21 total 60.03 last 62.06) total: 4500 open/close in 67.49 seconds: 66.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1331 (time 1787853323.81 total 10.00 last 133.05) - /unlink 2608 (time 1787853333.81 total 20.01 last 127.59) - /unlink 3903 (time 1787853343.82 total 30.01 last 129.48) total: 4500 /unlink in 34.91 seconds: 128.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 629 (time 1787853364.29 total 10.06 last 62.53) - open/close 1254 (time 1787853374.31 total 20.07 last 62.41) - open/close 1921 (time 1787853384.31 total 30.08 last 66.65) - open/close 2568 (time 1787853394.34 total 40.10 last 64.56) - open/close 3181 (time 1787853404.35 total 50.11 last 61.23) - open/close 3918 (time 1787853414.35 total 60.12 last 73.68) total: 4500 open/close in 67.61 seconds: 66.56 ops/second - /unlink 1369 (time 1787853436.77 total 10.00 last 136.87) - /unlink 2721 (time 1787853446.78 total 20.01 last 135.12) - /unlink 3951 (time 1787853456.78 total 30.01 last 122.99) total: 4500 /unlink in 34.37 seconds: 130.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 738 (time 1787853476.28 total 10.02 last 73.64) - open/close 1340 (time 1787853486.28 total 20.03 last 60.16) - open/close 1999 (time 1787853496.29 total 30.04 last 65.85) - open/close 2570 (time 1787853506.29 total 40.04 last 57.10) - open/close 3214 (time 1787853516.29 total 50.04 last 64.39) - open/close 3891 (time 1787853526.30 total 60.05 last 67.65) total: 4500 open/close in 68.34 seconds: 65.85 ops/second - /unlink 1357 (time 1787853549.58 total 10.01 last 135.63) - /unlink 2665 (time 1787853559.58 total 20.01 last 130.73) - /unlink 3944 (time 1787853569.60 total 30.02 last 127.75) total: 4500 /unlink in 35.73 seconds: 125.95 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95355 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server unloading modules via unload_modules_local on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: modules unloaded.