1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 3747 3748 3749 3750 3751 3752 3753 3754 3755 3756 3757 3758 3759 3760 3761 3762 3763 3764 3765 3766 3767 3768 3769 3770 3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 4066 4067 4068 4069 4070 4071 4072 4073 4074 4075 4076 4077 4078 4079 4080 4081 4082 4083 4084 4085 4086 4087 4088 4089 4090 4091 4092
|
#!/usr/bin/env bash
# Purpose: Climatology script tailored to E3SM (and hence CESM'ish) input and E3SM output guidelines
# Produces (and optionally regrids) climatological monthly means, seasonal means, annual means, CMIP6-style timeseries, global-mean timeseries, diurnal cycles
# Copyright (C) 2015--present Charlie Zender
# This file is part of NCO, the netCDF Operators. NCO is free software.
# You may redistribute and/or modify NCO under the terms of the
# 3-Clause BSD License.
# You are permitted to link NCO with the HDF, netCDF, OPeNDAP, and UDUnits
# libraries and to distribute the resulting executables under the terms
# of the BSD, but in addition obeying the extra stipulations of the
# HDF, netCDF, OPeNDAP, and UDUnits licenses.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the 3-Clause BSD License for more details.
# The original author of this software, Charlie Zender, seeks to improve
# it with your suggestions, contributions, bug-reports, and patches.
# Please contact the NCO project at http://nco.sf.net or write to
# Charlie Zender
# Department of Earth System Science
# University of California, Irvine
# Irvine, CA 92697-3100
# Prerequisites: Bash, NCO
# Script could use other shells, e.g., dash (Debian default) after rewriting function definitions and loops
# Debug with 'bash -x ncclimo --dbg=dbg_lvl' where 0 <= dbg_lvl <= 5
# Source: https://github.com/nco/nco/tree/master/data/ncclimo
# Documentation: http://nco.sf.net/nco.html#ncclimo
# Additional Documentation:
# ncclimo HowTo: https://e3sm.atlassian.net/wiki/spaces/DOC/pages/31129737/Generate+Regrid+and+Split+Climatologies+climo+files+with+ncclimo
# E3SM Climatology Requirements: https://acme-climate.atlassian.net/wiki/display/ATM/Climo+Files+-+v0.3+AMIP+runs
# Insta-install:
# scp ~/nco/data/ncclimo zender1@acme1.llnl.gov:bin
# scp ~/nco/data/ncclimo andes.olcf.ornl.gov:bin_andes
# scp ~/nco/data/ncclimo ac.zender@blues.lcrc.anl.gov:bin_blues
# scp ~/nco/data/ncclimo ac.zender@chrysalis.lcrc.anl.gov:bin_chrysalis
# scp ~/nco/data/ncclimo compy.pnl.gov:bin
# scp ~/nco/data/ncclimo derecho.hpc.ucar.edu:bin
# scp ~/nco/data/ncclimo dust.ess.uci.edu:bin
# scp ~/nco/data/ncclimo e3sm.ess.uci.edu:bin
# scp ~/nco/data/ncclimo imua.ess.uci.edu:bin
# scp ~/nco/data/ncclimo perlmutter-p1.nersc.gov:bin_perlmutter
# scp ~/nco/data/ncclimo spectral.ess.uci.edu:bin
# scp dust.ess.uci.edu:bin/ncclimo ~/bin
# scp dust.ess.uci.edu:bin/ncclimo ${MY_BIN_DIR}
# scp zender@dust.ess.uci.edu:bin/ncclimo ${MY_BIN_DIR}
# Set script name, directory, PID, run directory
drc_pwd=${PWD}
# Security: Explicitly unset IFS before wordsplitting, so Bash uses default IFS=<space><tab><newline>
unset IFS
# Set these before 'module' command which can overwrite ${BASH_SOURCE[0]}
# NB: dash supports $0 syntax, not ${BASH_SOURCE[0]} syntax
# http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
spt_src="${BASH_SOURCE[0]}"
[[ -z "${spt_src}" ]] && spt_src="${0}" # Use ${0} when BASH_SOURCE is unavailable (e.g., dash)
while [ -h "${spt_src}" ]; do # Recursively resolve ${spt_src} until file is no longer a symlink
drc_spt="$( cd -P "$( dirname "${spt_src}" )" && pwd )"
spt_src="$(readlink "${spt_src}")"
[[ ${spt_src} != /* ]] && spt_src="${drc_spt}/${spt_src}" # If ${spt_src} was relative symlink, resolve it relative to path where symlink file was located
done
cmd_ln="${spt_src} ${@}"
drc_spt="$( cd -P "$( dirname "${spt_src}" )" && pwd )"
spt_nm=$(basename ${spt_src}) # [sng] Script name (unlike $0, ${BASH_SOURCE[0]} works well with 'source <script>')
spt_pid=$$ # [nbr] Script PID (process ID)
# 20200110 Qi Tang reports his batch scripts on Cori that use netCDF4 map-files fail unless he sets this
if [ -z "${HDF5_USE_FILE_LOCKING}" ]; then
export HDF5_USE_FILE_LOCKING='FALSE'
fi # HDF5_USE_FILE_LOCKING
# 20221108: Passing environment variable NCO_PATH_OVERRIDE (NPO) to ncremap in batch queues fails on Cori
# Approaches that fail include: 1. export NPO='Yes';ncremap ... 2. export NPO='Yes' ncremap ...
# Direct approach that works is to pass NPO flag to ncremap via command-line switch
# Require that path-override switch to be first command-line option (i.e., $1) found with shell syntax
# ncremap/ncclimo implement NPO (though not getopt) logic prior to invoking NCO
# This switch is a no-op in main getopt() block below (since it has already been parsed here)
hrd_pth='No' # [sng] Hard-code machine-dependent paths/modules if HOSTNAME in database
if [ -n "${1}" ]; then
if [ "${1}" = '--hrd_pth' ] || [ "${1}" = '--npo' ] || [ "${1}" = '--nco_path_override' ] || [ "${1}" = '--NCO_PATH_OVERRIDE' ]; then
hrd_pth='Yes'
NCO_PATH_OVERRIDE='Yes'
fi # !hrd_pth
fi # !$1
# Configure paths at High-Performance Computer Centers (HPCCs) based on ${HOSTNAME}
if [ -z "${HOSTNAME}" ]; then
if [ -f /bin/hostname ] && [ -x /bin/hostname ]; then
export HOSTNAME=`/bin/hostname`
elif [ -f /usr/bin/hostname ] && [ -x /usr/bin/hostname ]; then
export HOSTNAME=`/usr/bin/hostname`
fi # !hostname
fi # HOSTNAME
# Default input and output directory is ${DATA}
if [ -z "${DATA}" ]; then
case "${HOSTNAME}" in
acme1* ) DATA="/home/${USER}" ; ;; # LLNL acme1
andes* ) DATA="/gpfs/alpine/world-shared/cli115/${USER}" ; ;; # OLCF andes compute nodes named andesNNN, 256 GB/node
blues* | blogin* | b[0123456789][0123456789][0123456789] ) DATA="/lcrc/project/ACME/${USER}" ; ;; # ANL/LCRC blues compute nodes named bNNN, 36|64 cores|GB/node
chrysalis* | chrlogin* | chr-[0123456789][0123456789][0123456789][0123456789] ) DATA="/lcrc/project/ACME/${USER}" ; ;; # ANL/LCRC chrysalis compute nodes named chr-NNNN, 64|256 cores|GB/node
compy* | n[0123456789][0123456789][0123456789][0123456789] ) DATA="/qfs/people/${USER}/data" ; ;; # PNNL compy compute nodes all named nNNNN, 40|192 cores|GB/node (compy login nodes also 192 GB)
constance* | node* ) DATA='/scratch' ; ;; # PNNL
derecho* ) DATA="/glade/p/work/${USER}" ; ;; # NCAR derecho compute nodes named, e.g., r8i0n8, r5i3n16, r12i5n29 ... 18|(64/256) cores|GB/node (derecho login nodes 512 GB)
frontier* ) DATA="/lustre/orion/cli115/world-shared/${USER}" ; ;; # OLCF frontier compute nodes named frontier01276 64|512 cores|GB/node
login[0123456789][0123456789] ) # 20230831 Frontier and Perlmutter login nodes share this name :(
if [ "${LMOD_SYSTEM_NAME}" = 'frontier' ]; then
DATA="/lustre/orion/cli115/world-shared/${USER}"
elif [ "${LMOD_SYSTEM_NAME}" = 'perlmutter' ]; then
DATA="${SCRATCH}"
fi # !LMOD_SYSTEM_NAME
;; # !login
perlmutter* | nid[0123456789][0123456789][0123456789][0123456789][0123456789][0123456789] ) DATA="${SCRATCH}" ; ;; # NERSC perlmutter compute nodes named nidNNNNNN (CPU) with (64)|(512) cores|GB/node (cpu) (login nodes 512 GB)
* ) DATA='/tmp' ; ;; # Other
esac # !HOSTNAME
fi # DATA
# 20190423 Speed-up OpenMP processes on Cori KNL Intel builds (and possibly others)
# Environmental settings (e.g., OMP_PROC_BIND=spread or KMP_PROC_BIND=intel) may place all threads on same hardware core
# Problem only known to manifest when multiple instances of NCO are spawned on single node
OMP_PROC_BIND=false
# Ensure batch jobs access correct 'mpirun' (or, with SLURM, 'srun') command, netCDF library, and NCO executables and library
# 20170914 Entire block is identical between ncclimo and ncremap---keep it that way!
# 20190421 Change override default from opt-out to opt-in
# 20221108 Implement hrd_pth in block above prior to getopt()
# 20250903 Simplify code: Always define then re-use E3SMU_ROOT, then add E3SMU_ROOT/bin to path only for ncremap
# Leave NCO_PATH_OVERRIDE unset or set to 'No' to prevent NCO from executing next block that overrides PATH
# Set NCO_PATH_OVERRIDE to 'Yes' in environment to cause NCO to execute next block and to override PATH:
# export NCO_PATH_OVERRIDE='Yes'
if [ "${hrd_pth}" = 'Yes' ] && [ "${NCO_PATH_OVERRIDE}" = 'Yes' ]; then
# If HOSTNAME is not in database, change hrd_pth_fnd to 'No' in case-statement default fall-through
hrd_pth_fnd='Yes' # [sng] Machine-dependent paths/modules for HOSTNAME found in database
case "${HOSTNAME}" in
acme1* )
E3SMU_ROOT='/p/user_pub/e3sm_unified/envs/base/envs/e3sm_unified_latest'
export PATH='/home/zender1/bin'\:${PATH}
export LD_LIBRARY_PATH='/home/zender1/lib'\:${E3SMU_ROOT}/lib\:${LD_LIBRARY_PATH} ; ;;
andes* )
# 20190827: Must guarantee finding mpirun
source ${MODULESHOME}/init/sh # 20150607: PMC Ensures find module commands will be found
if [ -d '/ccs/proj/cli115/software/e3sm-unified/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/ccs/proj/cli115/software/e3sm-unified/e3smu_latest_for_nco/andes'
else
E3SMU_ROOT='/ccs/proj/cli115/software/e3sm-unified/base/envs/e3sm_unified_latest'
fi
export PATH='/ccs/home/zender/bin_andes'\:${PATH}
export LD_LIBRARY_PATH='/ccs/home/zender/lib_andes'\:${E3SMU_ROOT}/lib\:${LD_LIBRARY_PATH} ; ;;
blues* | blogin* | b[0123456789][0123456789][0123456789] )
if [ -d '/lcrc/soft/climate/e3sm-unified/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/lcrc/soft/climate/e3sm-unified/e3smu_latest_for_nco/anvil'
else
E3SMU_ROOT='/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_latest'
fi
export PATH='/home/zender/bin_blues'\:${PATH}
export LD_LIBRARY_PATH='/home/zender/lib_blues'\:${E3SMU_ROOT}/lib\:${LD_LIBRARY_PATH} ; ;;
chrysalis* | chrlogin* | chr-[0123456789][0123456789][0123456789][0123456789] )
if [ -d '/lcrc/soft/climate/e3sm-unified/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/lcrc/soft/climate/e3sm-unified/e3smu_latest_for_nco/chrysalis'
else
E3SMU_ROOT='/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_latest'
fi
export PATH='/home/ac.zender/bin_chrysalis'\:${PATH}
export LD_LIBRARY_PATH='/home/ac.zender/lib_chrysalis:/home/ac.zender/anaconda/lib'\:${E3SMU_ROOT}/lib\:${LD_LIBRARY_PATH} ; ;;
compy* | n[0123456789][0123456789][0123456789][0123456789] )
module purge
module load gcc/10.2.0
# 20210519: This script takes significant time (5-10 seconds) to load
# 20230914: Deprecate special MOAB paths, rely on E3SMU
# source /compyfs/software/mbtempest.envs.sh
if [ -d '/share/apps/E3SM/conda_envs/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/share/apps/E3SM/conda_envs/e3smu_latest_for_nco/compy'
else
E3SMU_ROOT='/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_latest'
fi
export PATH='/qfs/people/zender/bin:/qfs/people/zender/anaconda/bin'\:${PATH}
export LD_LIBRARY_PATH='/qfs/people/zender/lib:/qfs/people/zender/anaconda/lib'\:${LD_LIBRARY_PATH} ; ;;
derecho* )
# 20240107: Derecho support not yet tested in batch mode
if [ ${spt_nm} = 'ncremap' ]; then
# On cheyenne, module load ncl installs ERWG in /glade/u/apps/ch/opt/ncl/6.4.0/intel/17.0.1/bin (i.e., ${NCARG_ROOT}/bin)
module load ncl
fi # !ncremap
if [ -n "${NCARG_ROOT}" ]; then
export PATH="${PATH}:/glade/u/apps/ch/opt/ncl/6.6.2/gnu/8.3.0/bin"
fi # !NCARG_ROOT
export PATH='/glade/u/home/zender/bin'\:${PATH}
export LD_LIBRARY_PATH='/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/gcc/12.2.0/ok4t/lib:/glade/u/apps/derecho/23.06/spack/opt/spack/udunits/2.2.28/gcc/12.2.0/vls2/lib:/glade/u/apps/ch/opt/gsl/2.7/gnu/12.1.0/lib:/glade/u/home/zender/lib'\:${LD_LIBRARY_PATH} ; ;;
e3sm* )
export PATH='/home/zender/bin:/home/zender/anaconda/bin'\:${PATH}
export LD_LIBRARY_PATH='/home/zender/lib:/home/zender/anaconda/lib'\:${LD_LIBRARY_PATH} ; ;;
frontier* )
if [ ${spt_nm} = 'ncremap' ]; then
if [ -d '/ccs/proj/cli115/software/e3sm-unified/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/ccs/proj/cli115/software/e3sm-unified/e3smu_latest_for_nco/frontier'
else
E3SMU_ROOT='/ccs/proj/cli115/software/e3sm-unified/base/envs/e3sm_unified_latest'
fi
fi # !ncremap
export PATH='/ccs/home/zender/bin_frontier:/ccs/home/zender/anaconda/bin'\:${PATH}
export LD_LIBRARY_PATH='/ccs/home/zender/lib_frontier:/ccs/home/zender/anaconda/lib'\:${LD_LIBRARY_PATH} ; ;;
login[0123456789][0123456789] ) # 20230831 Frontier and Perlmutter login nodes share this name :(
if [ "${LMOD_SYSTEM_NAME}" = 'frontier' ]; then
if [ -d '/ccs/proj/cli115/software/e3sm-unified/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/ccs/proj/cli115/software/e3sm-unified/e3smu_latest_for_nco/frontier'
else
E3SMU_ROOT='/ccs/proj/cli115/software/e3sm-unified/base/envs/e3sm_unified_latest'
fi
export PATH='/ccs/home/zender/bin_frontier:/ccs/home/zender/anaconda/bin'\:${PATH}
export LD_LIBRARY_PATH='/ccs/home/zender/lib_frontier:/ccs/home/zender/anaconda/lib'\:${LD_LIBRARY_PATH}
elif [ "${LMOD_SYSTEM_NAME}" = 'perlmutter' ]; then
# 20221103 Add build environment modules
module load PrgEnv-gnu
module load cray-hdf5/1.12.2.9
module load cray-netcdf/4.9.0.9
MOAB_ROOT=/project/projectdirs/e3sm/software/moab
TEMPESTREMAP_ROOT=/project/projectdirs/e3sm/software/tempestremap
if [ -d '/global/common/software/e3sm/anaconda_envs/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/global/common/software/e3sm/anaconda_envs/e3smu_latest_for_nco/pm-cpu'
else
E3SMU_ROOT='/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_latest'
fi
if [ -n "${NCARG_ROOT}" ]; then
export PATH="${PATH}:${NCARG_ROOT}/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${NCARG_ROOT}/lib"
fi # !NCARG_ROOT
export PATH='/global/cfs/cdirs/e3sm/zender/bin_perlmutter'\:${PATH}
export LD_LIBRARY_PATH='/global/cfs/cdirs/e3sm/zender/lib_perlmutter:/global/cfs/cdirs/e3sm/zender/anaconda/lib'\:${LD_LIBRARY_PATH}
fi # !LMOD_SYSTEM_NAME
;; # !login
perlmutter* | nid[0123456789][0123456789][0123456789][0123456789][0123456789][0123456789] )
# 20221103 Add build environment modules
module load PrgEnv-gnu
module load cray-hdf5/1.12.2.9
module load cray-netcdf/4.9.0.9
MOAB_ROOT=/project/projectdirs/e3sm/software/moab
TEMPESTREMAP_ROOT=/project/projectdirs/e3sm/software/tempestremap
if [ -d '/global/common/software/e3sm/anaconda_envs/e3smu_latest_for_nco' ]; then
E3SMU_ROOT='/global/common/software/e3sm/anaconda_envs/e3smu_latest_for_nco/pm-cpu'
else
E3SMU_ROOT='/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_latest'
fi
if [ -n "${NCARG_ROOT}" ]; then
export PATH="${PATH}:${NCARG_ROOT}/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${NCARG_ROOT}/lib"
fi # !NCARG_ROOT
export PATH='/global/cfs/cdirs/e3sm/zender/bin_perlmutter'\:${PATH}
export LD_LIBRARY_PATH='/global/cfs/cdirs/e3sm/zender/lib_perlmutter:/global/cfs/cdirs/e3sm/zender/anaconda/lib'\:${E3SMU_ROOT}/lib\:${LD_LIBRARY_PATH} ; ;;
* ) # Default fall-through
hrd_pth_fnd='No' ; ;;
esac # !HOSTNAME
if [ -n "${MOAB_ROOT}" ] && [ ${spt_nm} = 'ncremap' ]; then
export PATH="${PATH}:${MOAB_ROOT}/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${MOAB_ROOT}/lib"
fi # !MOAB_ROOT
if [ -n "${TEMPESTREMAP_ROOT}" ] && [ ${spt_nm} = 'ncremap' ]; then
export PATH="${PATH}:${TEMPESTREMAP_ROOT}/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${TEMPESTREMAP_ROOT}/lib"
fi # !TEMPESTREMAP_ROOT
# 20210519: E3SM-U supplies many uncommon executables and libraries, e.g., libantlr, ESMF_RegridWeightGen
# 20241220: Now that ncclimo also regularly uses ncap2 executable for global timeseries, must modify its path too
# ncclimo only needs access to E3SM_U libraries (e.g., libgsl, libantlr), already implemented above
# ncremap needs access to E3SM_U binaries (ESMF_RegridWeightGen, ...) _and_ libraries
# Append E3SM-U binaries to _end_ of PATH so NCO binaries not redirected, e.g., from CSZ's development directory to E3SM-U
if [ -n "${E3SMU_ROOT}" ] && [ ${spt_nm} = 'ncremap' ]; then
export PATH="${PATH}:${E3SMU_ROOT}/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${E3SMU_ROOT}/lib"
fi # !E3SMU_ROOT
fi # !hrd_pth && !NCO_PATH_OVERRIDE
# Test cases ($DATA/[grids/maps] refers to ~zender/data/[grids/maps] on Charlie's test machines)
# Production usage:
# ncclimo -P cam -c famipc5_ne30_v0.3_00003 -s 1979 -e 1983 -r ${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -O ${DATA}/ne30/rgr # LR
# ncclimo -P cam -c fxm -s 1 -e 127 -i -o ${DATA}/ne120/clm # HR
# Incremental climo testing:
# ncclimo -v FSNT,AODVIS -c famipc5_ne30_v0.3_00003 -s 1980 -e 1981 -i ${DATA}/ne30/raw -o ${DATA}/ne30/prv -r ${DATA}/maps/map_ne30np4_to_fv129x256_aave.20150901.nc
# ncclimo -v FSNT,AODVIS -c famipc5_ne30_v0.3_00003 -s 1982 -e 1983 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -r ${DATA}/maps/map_ne30np4_to_fv129x256_aave.20150901.nc -x ${DATA}/ne30/prv -X ${DATA}/ne30/xtn -S 1980
# Binary climo testing:
# ncclimo -v FSNT,AODVIS -c famipc5_ne30_v0.3_00003 -S 1980 -E 1981 -x ${DATA}/ne30/prv -s 1982 -e 1983 -i ${DATA}/ne30/clm -X ${DATA}/ne30/xtn
# Annual climo testing:
# ncclimo -C ann -m cism -h h -c b.e10.BG20TRCN.f09_g16.002 -s 1851 -e 2006 -i /gpfs/alpine/cli115/proj-shared/4ue/data/for_charlie -o ${DATA}/ne30/clm
# ncclimo -C ann -m cism -h h -c b.e10.BG20TRCN.f09_g16.002 -s 1851 -e 1852 -i /gpfs/alpine/cli115/proj-shared/4ue/data/for_charlie -o ${DATA}/ne30/clm > ~/ncclimo.out 2>&1 &
# ncclimo -C ann -m mpas -s 1851 -e 1852 -i ${DATA}/hdf/mpasLIoutput.nc -o ${DATA}/mpas/clm > ~/ncclimo.out 2>&1 &
# High-frequency climos:
# drc_in=${DATA}/ne30/raw
# caseid=famipc5_ne30_v0.3_00007.cam.h4 # 3-hour resolution
# cd ${drc_in};ls ${caseid}*.nc | ncclimo --var=PRECT --clm_md=hfc --caseid=hfc --yr_srt=2001 --yr_end=2002 --drc_out=${HOME} --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc # Multi-file input
# ncclimo --var=PRECT --clm_md=hfc --caseid=hfc --yr_srt=2001 --yr_end=2002 --drc_out=${HOME} --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc ~/PRECT_200101_200212.nc # Single file input
# Debugging and Benchmarking:
# E3SM v3:
# ncclimo -7 --cmp='gbr|shf|zst' -P eam -v FSNT,AODVIS,TREFHT -c v3.LR.piControl -s 460 -e 461 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -O ${DATA}/ne30/rgr -r ${DATA}/maps/map_ne30pg2_to_cmip6_180x360_traave.20231201.nc # Compression!
# E3SM v2:
# ncclimo -7 --cmp='gbr|shf|zst' -P eam -v FSNT,AODVIS,TREFHT -c v2.LR.historical_0101 -s 2013 -e 2014 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -O ${DATA}/ne30/rgr -r ${DATA}/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc # Compression!
# ncclimo -P eam -v FSNT,AODVIS,TREFHT -c v2.LR.historical_0101 -s 2013 -e 2014 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -O ${DATA}/ne30/rgr -r ${DATA}/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc
# cd ${DATA}/ne30/raw;ls v2.LR*eam*201[34]-??*.nc | ncclimo --split --dbg=1 --rgn_avg -s 2013 -e 2014 --var=TREFHT,FSNT,AODVIS --drc_out=${DATA}/ne30/clm
# cd ${DATA}/ne30/raw;ls v2.LR*elm*201[34]-??*.nc | ncclimo -P elm --split --dbg=1 --rgn_avg -s 2013 -e 2014 --var=FSDS,TBOT --drc_out=${DATA}/ne30/clm
# ncclimo -P eam --var_lst=FSNT,AODVIS,TREFHT --caseid=v2.LR.historical_0101 --yr_srt=2013 --yr_end=2014 --drc_in=${DATA}/ne30/raw --drc_out=${DATA}/ne30/clm --map=${DATA}/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc
# E3SM v1:
# ncclimo -P eam -m cam -v FSNT,AODVIS,TREFHT -c 20180129.DECKv1b_piControl.ne30_oEC.edison -s 1 -e 2 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -O ${DATA}/ne30/rgr -r ${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc
# ncclimo -P eam -m cam --var_lst=FSNT,AODVIS,TREFHT --caseid=20180129.DECKv1b_piControl.ne30_oEC.edison --yr_srt=1 --yr_end=2 --drc_in=${DATA}/ne30/raw --drc_out=${DATA}/ne30/clm --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc
# ncclimo -v TOTEXTTAU -c merra2_198001.nc4 -s 1980 -e 2015 -i ${DATA}/merra2/raw -o ${DATA}/merra2/clm
# ncclimo > ~/ncclimo.out 2>&1 &
# ncclimo -c B1850C5e1_ne30 -s 2 -e 199 > ~/ncclimo.out 2>&1 &
# ncclimo -c ne30_gx1.B1850c5d -s 6 -e 7 > ~/ncclimo.out 2>&1 &
# ncclimo -d 2 -v FSNT -m cam2 -c essgcm14 -s 1 -e 20 -i ${DATA}/essgcm14 -o ${DATA}/anl > ~/ncclimo.out 2>&1 &
# MPAS: Prior to running ncclimo on MPAS output, annotate missing values of input with, e.g.,
# for fl in `ls hist.*` ; do
# ncatted -O -t -a _FillValue,,o,d,-9.99999979021476795361e+33 ${fl}
# done
# New MPAS filename conventions (as of ~201612):
# ncclimo --wnt_md=jfd --vars=timeMonthly_avg_iceAreaCell --yr_srt=2 --yr_end=3 --mdl=mpasseaice --d2f --drc_in=/p/user_pub/work/E3SM/1_0/piControl/1deg_atm_60-30km_ocean/sea-ice/native/model-output/mon/ens1/v1 --map=${DATA}/maps/map_oEC60to30v3_to_cmip6_180x360_aave.20181001.nc --drc_out=${DATA}/ne30/clm --drc_rgr=${DATA}/ne30/rgr > ~/ncclimo.out 2>&1 &
# ncclimo --wnt_md=jfd --vars=timeMonthly_avg_activeTracers_temperature --yr_srt=2 --yr_end=3 --mdl=mpaso --d2f --dpt_fl=${DATA}/grids/mpas_refBottomDepth_60lyr.nc --drc_in=/p/user_pub/work/E3SM/1_0/piControl/1deg_atm_60-30km_ocean/ocean/native/model-output/mon/ens1/v1 --map=${DATA}/maps/map_oEC60to30v3_to_cmip6_180x360_aave.20181001.nc --drc_out=${DATA}/ne30/clm --drc_rgr=${DATA}/ne30/rgr > ~/ncclimo.out 2>&1 &
# Old MPAS filename conventions (until ~201609)::
# ncclimo -v temperature -c hist -s 2 -e 3 -m ocn -i /gpfs/alpine/cli112/proj-shared/golaz/ACME_simulations/20160121.A_B2000ATMMOD.ne30_oEC.titan.a00/run -r ${DATA}/maps/map_oEC60to30_to_t62_bilin.20160301.nc -o ${DATA}/mpas/clm > ~/ncclimo.out 2>&1 &
# ncclimo -v iceAreaCell -c hist -s 2 -e 3 -m ice -i /gpfs/alpine/cli112/proj-shared/golaz/ACME_simulations/20160121.A_B2000ATMMOD.ne30_oEC.titan.a00/run -r ${DATA}/maps/map_oEC60to30_to_t62_bilin.20160301.nc -o ${DATA}/mpas/clm > ~/ncclimo.out 2>&1 &
# Split pipe:
# cd ${DATA}/ne30/raw;ls *000[12]-??*.nc | ncclimo --split --dbg=1 -s 1 -e 2 --var=FSNT,AODVIS --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_out=${DATA}/ne30/clm
# Split redirection:
# cd ${DATA}/ne30/raw;ls *000[12]-??*.nc > ~/foo;ncclimo --split --dbg=1 -s 1 -e 2 --var=FSNT,AODVIS --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_out=${DATA}/ne30/clm < ~/foo
# Split stdin switch:
# cd ${DATA}/ne30/raw;ls *000[12]-??*.nc | ncclimo --split --dbg=1 --stdin -s 1 -e 2 --var=FSNT,AODVIS --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_out=${DATA}/ne30/clm
# Split positional arguments:
# ncclimo --split --dbg=1 -s 1 -e 1 --var=FSNT,AODVIS,T --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_out=${DATA}/ne30/clm ${DATA}/ne30/raw/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0001-??.nc
# Split directory:
# ncclimo --dbg=1 --ypf=50 -s 1 -e 2 --var=FSNT,AODVIS --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_in=${DATA}/ne30/raw --drc_out=${DATA}/ne30/clm
# Split global mean
# cd ${DATA}/ne30/raw;ls 2018*000[12]-??*.nc | ncclimo --split --dbg=1 --rgn_avg -s 1 -e 2 --var=FSNT,AODVIS,TREFHT --drc_out=${DATA}/ne30/clm
# Split global sum
# ncclimo -P elm --split --rgn_stt=sum --sum_scl=1.0e6*3600*24*365/1.0e12 -v GPP -c v3.LR.piControl -s 460 -e 461 --var=GPP --drc_in=${DATA}/ne30/raw --drc_out=${DATA}/ne30/clm
# Split all:
# cd ${DATA}/ne30/raw;ls *000[12]-??*.nc | ncclimo --split --dbg=1 -s 1 -e 2 --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_out=${DATA}/ne30/clm
# Split ELM:
# ncclimo --split --dbg=1 -s 2000 -e 2000 --var=FSDS,TBOT --drc_out=${DATA}/ne30/clm ${DATA}/ne30/raw/F_acmev03_enso_camse_clm45bgc_ne30_co2cycle.clm2.h0.2000-??.nc
# Split vertical:
# cd ${DATA}/ne30/raw;ls *000[12]-??*.nc | ncclimo --split --dbg=1 -s 1 -e 2 --var=T --vrt_out=${DATA}/grids/vrt_prs_ncep_L17.nc --drc_out=${DATA}/ne30/clm
# cd ${DATA}/ne30/raw;ls *000[12]-??*.nc | ncclimo --split --dbg=1 -s 1 -e 2 --var=T --vrt_out=${DATA}/grids/vrt_prs_ncep_L17.nc --vrt_xtr=mss_val --drc_out=${DATA}/ne30/clm # Missing value interpolation
# Split production:
# cd /scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850S.ne30_oEC_ICG.edison/run;ls *cam.h0.000[1-9]* | ncclimo --split --dbg=1 --yr_srt=1 --yr_end=9 --var=FSNT,AODVIS,T --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_out=${DATA}/ne30/clm > ~/ncclimo.out 2>&1 &
# cd /scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850S.ne30_oEC_ICG.edison/run;ls *cam.h0.0[012]??* | ncclimo --split --dbg=1 --yr_srt=1 --yr_end=250 --var=FSNT,AODVIS,T --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc --drc_out=${DATA}/ne30/clm > ~/ncclimo.out 2>&1 &
# Daily pipe:
# cd ${DATA}/ne30/raw;ls *h1*.nc | ncclimo --dbg=1 --job_nbr=8 --caseid=famipc5_ne30_v0.3_00007 --clm_md=dly -s 2001 -e 2002 --var=PRECT,TREFHT --drc_out=${DATA}/ne30/clm > ~/ncclimo.out 2>&1 &
# Diurnal pipe:
# cd ${DATA}/ne30/raw;ls *h4*.nc | ncclimo --dbg=1 --job_nbr=8 --caseid=famipc5_ne30_v0.3_00007 --clm_md=dly --tpd=8 -s 2001 -e 2002 --var=PRECT,TS,FSDS --drc_out=${DATA}/ne30/clm > ~/ncclimo.out 2>&1 &
# SGS:
# ncclimo -P elm -m clm2 -d 1 -v TBOT,FSDS -c essgcm14 -s 1 -e 1 -i ${DATA}/essgcm14 -o ${DATA}/ne30/rgr --map=${DATA}/maps/map_t42_to_fv129x256_aave.20150901.nc --sgs_frc=${DATA}/grids/clm2_landfrac_t42.nc/landfrac
# ncclimo -P elm -m clm2 -d 1 -v TBOT,FSDS -c F_acmev03_enso_camse_clm45bgc_ne30_co2cycle -s 2000 -e 2000 -i ${DATA}/ne30/raw -o ${DATA}/ne30/rgr --map=${DATA}/maps/map_ne30np4_to_cmip6_180x360_aave.20181001.nc --sgs_frc=${DATA}/grids/elm_landfrac_ne30np4.nc/landfrac
# ncclimo -P elm -d 1 -v TBOT,FSDS -c v2.LR.historical_0101 -s 2013 -e 2014 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -O ${DATA}/ne30/rgr --map=${DATA}/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc --sgs_frc=${DATA}/grids/elm_landfrac_ne30pg2.nc/landfrac
# Seasons:
# ncclimo -d 1 --csn_lst=mam,jja,son,djf,jfm,on -v FSNT -c 20180129.DECKv1b_piControl.ne30_oEC.edison -s 1 -e 2 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -r ${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc
# ncclimo -d 1 --csn_lst=jfm,jas,ann -v FSNT -c 20180129.DECKv1b_piControl.ne30_oEC.edison -s 1 -e 2 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -r ${DATA}/maps/map_ne30np4_to_cmip6_180x360_nco.20190601.nc
# Compy Jobs or MPI:
# ncclimo -c 20180215.DECKv1b_H1.ne30_oEC.edison -s 2010 -e 2014 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -O ${DATA}/ne30/rgr -r ${DATA}/maps/map_ne30np4_to_cmip6_180x360_aave.20181001.nc # ne30 climo on 1 node in full-background (12 jobs) mode
# ncclimo --jobs=3 -c 20180410.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta -s 1 -e 5 -i ${DATA}/ne120/raw -o ${DATA}/ne120/clm -r ${DATA}/maps/map_ne120np4_to_cmip6_720x1440_aave.20181001.nc # ne120 climo on 1 node in background mode with 3-jobs (to stay under 192 GB RAM limit)
# echo '#!/bin/bash' > ~/ncclimo.slurm
# echo "ncclimo -p mpi -c 20180410.A_WCYCL1950_HR.ne120_oRRS18v3_ICG.theta -s 1 -e 5 -i ${DATA}/ne120/raw -o ${DATA}/ne120/clm -O ${DATA}/ne120/rgr -r ${DATA}/maps/map_ne120np4_to_cmip6_720x1440_aave.20181001.nc" >> ~/ncclimo.slurm;chmod a+x ~/ncclimo.slurm # ne120 climo on 4 nodes, MPI mode:
# sbatch -A e3sm --nodes=4 --time=00:30:00 --job-name=ncclimo --mail-type=fail,end --mail-user=zender@uci.edu --output=${HOME}/ncclimo.out ~/ncclimo.slurm
# Best performance on resolutions finer than ne30 (~1x1 degree) requires a job scheduler/batch processor
# SLURM (andes,compy,cori), Maui (a PBS-variant) (blues), Torque (a PBS-variant) (hopper), and PBS (other) schedulers allow both interactive and non-interactive (i.e., script) batch jobs
# ALCF Maui:
# http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs
# NCAR PBSPro:
# https://arc.ucar.edu/knowledge_base/124518554
# https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne/quick-start-cheyenne
# NERSC SLURM:
# https://slurm.schedmd.com/sbatch.html # sbatch man page
# https://slurm.schedmd.com/salloc.html # salloc man page
# OLCF SLURM (after 20190903, PBS until 20190903):
# https://docs.olcf.ornl.gov/systems/andes_user_guide.html
# https://docs.olcf.ornl.gov/systems/frontier_user_guide.html
# Requesting interactive nodes, Submitting non-interactive batch jobs, Monitoring queues, Deleting jobs:
# LSF: bsub -I, bsub,jobstat, qdel, jsrun
# PBS: qsub -I, qsub, qstat, qdel, mpirun
# PBSPro: qsub -I, qsub, qstat, qdel, mpirun
# SLURM: salloc, sbatch, squeue, scancel, srun
# Interactive queue: a) Reserve nodes and acquire prompt on control node b) Execute ncclimo command interactively
# Andes: salloc -A CLI115 --nodes=1 --time=00:30:00 --job-name=ncclimo # Bigmem: --partition=gpu
# Blues: salloc -A condo --nodes=1 --partition=acme-small --time=00:30:00 --job-name=ncclimo
# Chrysalis: salloc -A e3sm --nodes=1 --partition=debug --time=00:30:00 --job-name=ncclimo # Use compute queue instead if debug is slow to grant resources
# Compy: salloc -A e3sm --nodes=1 --time=00:30:00 --job-name=ncclimo
# Derecho: qsub -I -A SCSG0002 -l select=1:ncpus=36:mpiprocs=1 -l walltime=00:30:00 -q regular -N ncclimo
# Frontier:salloc -A CLI115 --nodes=1 --time=00:30:00 --job-name=ncclimo
# Perl: salloc -A e3sm --constraint=cpu --nodes=1 --partition=debug --time=00:30:00 --job-name=ncclimo # NB: 30 minute limit
# Non-interactive batch procedure: a) Store ncclimo command in ncclimo.[pbs|slurm] b) qsub ncclimo.[pbs|slurm]
# Non-interactive batch queue differences (besides argument syntax):
# 1. SLURM requires initial 'shebang' line to specify the shell interpreter (not required on PBS)
# 2. PBS overwrites existing files
# 3. PBS (and PBSPro) use ${PBS_NODEFILE} and ${PBS_NUM_PPN}, respectively, and SLURM uses ${SLURM_NODELIST} and ${SLURM_CPUS_ON_NODE}, respectively
# 4. SLURM automatically combines stdout and stderr, yet does not understand tilde (~ = home directory) expansion in error/output filenames
# Differences 1 & 2 impose slightly different invocations; difference 3 requires abstracting environment variables; difference 4 requires omitting ~'s
# Andes a): echo '#!/bin/bash' > ~/ncclimo.slurm
# echo "ncclimo -a djf -d 1 -p mpi -c famipc5_ne120_v0.3_00003 -s 1 -e 2 -i ${DATA}/ne120/raw -o ${DATA}/ne120/clm -r ${DATA}/maps/map_ne120np4_to_fv257x512_aave.20150901.nc" >> ~/ncclimo.slurm;chmod a+x ~/ncclimo.slurm
# Blues a): echo "ncclimo -a djf -d 1 -p mpi -c 20180129.DECKv1b_piControl.ne30_oEC.edison -s 1 -e 2 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -r ${DATA}/maps/map_ne30np4_to_fv129x256_aave.20150901.nc" > ~/ncclimo.pbs;chmod a+x ~/ncclimo.pbs
# Cheye a): echo "ncclimo -a djf -d 1 -p mpi -c 20180129.DECKv1b_piControl.ne30_oEC.edison -s 1 -e 2 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -r ${DATA}/maps/map_ne30np4_to_fv129x256_aave.20150901.nc" > ~/ncclimo.pbs;chmod a+x ~/ncclimo.pbs
# Compy a): echo '#!/bin/bash' > ~/ncclimo.slurm
# echo "ncclimo -d 1 -p mpi -c 20180215.DECKv1b_H1.ne30_oEC.edison -s 2010 -e 2010 -i ${DATA}/ne30/raw -o ${DATA}/ne30/clm -r ${DATA}/maps/map_ne30np4_to_fv129x256_aave.20150901.nc" >> ~/ncclimo.slurm;chmod a+x ~/ncclimo.slurm
# Front a): echo '#!/bin/bash' > ~/ncclimo.slurm
# echo "ncclimo -a djf -d 1 -p mpi -c famipc5_ne120_v0.3_00003 -s 1 -e 2 -i ${DATA}/ne120/raw -o ${DATA}/ne120/clm -r ${DATA}/maps/map_ne120np4_to_fv257x512_aave.20150901.nc" >> ~/ncclimo.slurm;chmod a+x ~/ncclimo.slurm
# Andes b): sbatch -A CLI115 --nodes=12 --time=00:30:00 --job-name=ncclimo --mail-type=fail,end --mail-user=zender@uci.edu --output=${HOME}/ncclimo.out ~/ncclimo.slurm
# Blues b): qsub -A ACME -q acme -l nodes=1 -l walltime=00:30:00 -N ncclimo -j oe -m e -o ~/ncclimo.out ~/ncclimo.pbs
# Cheye b): qsub -A SCSG0002 -l select=12:ncpus=36:mpiprocs=1 -l walltime=00:30:00 -q regular -N ncclimo -j oe -m e -o ~/ncclimo.out ~/ncclimo.pbs
# Compy b): sbatch -A e3sm --nodes=12 --time=00:30:00 --job-name=ncclimo --mail-type=fail,end --mail-user=zender@uci.edu --output=${HOME}/ncclimo.out ~/ncclimo.slurm
# Front b): sbatch -A CLI115 --nodes=12 --time=00:30:00 --job-name=ncclimo --mail-type=fail,end --mail-user=zender@uci.edu --output=${HOME}/ncclimo.out ~/ncclimo.slurm
# Perlm b): sbatch -A e3sm --constraint=cpu --nodes=12 --time=00:30:00 --qos=debug --job-name=ncclimo --mail-type=fail,end --mail-user=zender@uci.edu --output=${HOME}/ncclimo.out ~/ncclimo.slurm
# NB: stdout/stderr files returned from batch jobs are read-protected, need 'chmod 644 ~/ncclimo.out' afterwards
# Normal use: Set five "mandatory" inputs (caseid, yr_srt, yr_end, drc_in, drc_out), and possibly rgr_map, on command line
# caseid: Simulation name (filenames must start with ${caseid})
# drc_in: Input directory for raw data
# Years outside yr_srt and yr_end are ignored
# yr_srt should, and for JFD mode must, contain complete year of output
# DJF mode ignores Jan-Nov of yr_srt
# Dec of yr_end is excluded from the seasonal and monthly analysis in DJF mode
# yr_end should, and for JFD mode must, contain complete year of output
# drc_out: Output directory for processed native grid climatology ("climo files")
# User needs write permission for ${drc_out}
# rgr_map: Horizontal regridding map, if non-NULL, invoke regridder with specified map on output datasets
# Pass options intended exclusively for the NCO regridder as arguments to the -R switch
# yr_srt: Year of first January to analyze
# yr_end: Year of last January to analyze
# Other options (often their default settings work well):
# wnt_md: Winter mode aka December mode, i.e., how to treat December in (NH) Winter seasons. One of two options:
# Seasonally-contiguous-december (SCD or DJF) mode (wnt_md=scd|djf) (default through 4.9.4)
# Seasonally-discontiguous-december (SDD or JFD) mode (wnt_md=sdd|jfd) (default starting 4.9.5)
# Both modes use an integral multiple of 12 months, and _never alter any input files_
# DJF climatologies begin in Dec of yr_srt-1, and end in Nov of yr_end
# JFD climatologies begin in Jan of yr_srt, and end in Dec of yr_end
# DJF excludes Jan-Nov of yr_srt-1 and Dec of yr_end (i.e., DJF excludes 12 months of available data)
# JFD uses all months of yr_srt through yr_end (i.e., JFD can use all available data)
# DJF seasonal averages are inconsistent with (calendar-year-based) annual averages, but better capture seasonal the "natural" (not calendar-year-based) climate year
# JFD seasonal averages are fully consistent with (calendar-year-based) annual averages
# drc_rgr: Regridding directory---store regridded files, if any, in drc_rgr rather than drc_out
# lnk_flg: Link E3SM-climo to AMWG-climo filenames
# AMWG omits the YYYYMM components of climo filenames, resulting in shorter names
# This switch (on by default) symbolically links the full (E3SM) filename to the shorter (AMWG) name
# AMWG diagnostics scripts can produce plots directly from these linked filenames
# par_typ: Parallelism type for files
# bck = Background: Spawn children (basic blocks) as background processes on control node then wait()
# Works best when available RAM > 12*4*sizeof(monthly input file), otherwise jobs swap-to-disk
# mpi = MPI: Spawn children (basic blocks) as MPI processes (one per node in batch environment) then wait()
# Requires batch system with PBS and MPI. Use when available RAM/node < 12*2.5*sizeof(monthly input file).
# Optimized for batch with 12 nodes. Factors thereof (6, 4, 3, 2 nodes) should also work.
# Remember to request 12 nodes if possible!
# srl = Serial: Execute script in serial mode on single node
# Works best when available RAM < 12*4*sizeof(monthly input file), otherwise jobs swap-to-disk
# var_lst: Variables to include, or, with nco_opt='-x', to exclude, in comma-separated list format, e.g.,
# 'FSNT,AODVIS'. Regular expressions work, too: 'AOD.?'
# Infrequently used options:
# dbg_lvl: 0 = Quiet, print basic status during evaluation
# 1 = Print configuration, full commands, and status to output during evaluation
# 2 = As in dbg_lvl=1, but do not evaluate commands
# 3 = As in dbg_lvl=2, with additional information (mainly for batch queues)
# fml_nm: Family name (nickname) of output files referring to $fml_nm character sequence used in output climo file names:
# fml_nm_XX_YYYYMM_YYYYMM.nc (examples include '' (default), 'control', 'experiment')
# By default, fml_nm=$caseid. Use fml_nm instead of $caseid to simplify long names, avoid overlap, etc.
# hst_nm: History volume name string, i.e., $hst_nm character sequence used in history tape names:
# caseid.mdl_nm.hst_nm.YYYY-MM.nc (examples include 'h0' (default, works for cam, clm), 'h1', 'h' (for cism), 'hist' (for mpasocean, mpasseaice)
# mdl_nm: Model name string in output filenames, i.e., character sequence $mdl_nm in standard names:
# caseid.mdl_nm.h0.YYYY-MM.nc (examples include 'cam' (default), 'clm2', 'cam2', 'cice', 'cism', 'eam', 'elm', 'mosart', 'mpasocean', 'mpaso', 'mpasseaice', 'mpassi', 'pop')
# nco_opt: String of options to pass-through to NCO, e.g.,
# '-D 2 -7 -L 1' for NCO debugging level 2, netCDF4-classic output, compression level 1
# '--no_tmp_fl -x' to skip temporary files, turn extraction into exclusion list
# rgr_opt: String of options (besides thread-number) to pass-through exclusively to NCO regridder, e.g.,
# ncclimo -m clm2 ... -R col_nm=lndgrid -r map.nc ...
# thr_nbr: Thread number to use in NCO regridder, '-t 1' for one thread, '-t 2' for two threads...
# 20220131 ncclimo/ncremap commands within scripts that open E3SMU environment
# (e.g., zppy-generated scripts) must be told where to find NCO binaries.
# Only necessary on login nodes since Spack handles this fine on compute nodes
if [ "${E3SMU_MPI}" = 'NOMPI' ] && [ -n "${E3SMU_SCRIPT}" ] && [ -n "${CONDA_PREFIX}" ]; then
export PATH="${CONDA_PREFIX}/bin"\:${PATH}
fi # !E3SMU_MPI
# Set NCO version and directory
nco_exe=`which ncks`
if [ -z "${nco_exe}" ]; then
echo "${spt_nm}: ERROR Unable to find NCO, \${nco_exe} = ${nco_exe}"
echo "${spt_nm}: HINT Carefully examine your environment setup (e.g., .bashrc) to avoid inadvertently overriding (with, e.g., conda-initialization) paths intended to be provided by an analysis-package environment (e.g., E3SM-Unified)"
exit 1
fi # !nco_exe
# StackOverflow method finds NCO directory
while [ -h "${nco_exe}" ]; do
drc_nco="$( cd -P "$( dirname "${nco_exe}" )" && pwd )"
nco_exe="$(readlink "${nco_exe}")"
[[ ${nco_exe} != /* ]] && nco_exe="${drc_nco}/${nco_exe}"
done
drc_nco="$( cd -P "$( dirname "${nco_exe}" )" && pwd )"
nco_vrs=$(ncks --version 2>&1 > /dev/null | grep NCO | awk '{print $5}')
# 20190218: Die quickly when NCO is found yet cannot run, e.g., due to linker errors
if [ -z "${nco_vrs}" ]; then
echo "${spt_nm}: ERROR ${nco_exe} dies with error message on next line:"
$(ncks --version)
# 20250905: Additional information for debugging link issues
if [ "${NCO_PATH_OVERRIDE}" = 'Yes' ]; then
echo "${spt_nm}: DEBUG Next line shows contents of \$LD_LIBRARY_PATH:"
echo "${LD_LIBRARY_PATH}"
echo "${spt_nm}: DEBUG Next line shows contents of \$PATH:"
echo "${PATH}"
else # !NCO_PATH_OVERRIDE
printf "HINT: Run-time error due to link issues (e.g., libraries missing or not found) might be solved at supported national labs (ALCF, NCAR, NERSC, OLCF, PNNL) by employing machine-dependent hardcoded paths/modules. To try this, re-run command with \"--npo\" as first option, i.e., \"${spt_src} --npo ...\" or after setting \"export NCO_PATH_OVERRIDE=Yes\".\n"
fi # !NCO_PATH_OVERRIDE
exit 1
fi # !nco_vrs
nco_sng=$(ncks --version 2>&1 > /dev/null | grep NCO | awk -F '"' '{print $2}')
lbr_vrs=$(ncks --library 2>&1 > /dev/null | awk '{print $6}')
# Detect and warn about mixed modules (for Qi Tang 20170531)
if [ "${drc_spt}" != "${drc_nco}" ]; then
echo "INFO: Mixture of NCO scripts and binaries from different locations. Script ${spt_nm} is from directory ${drc_spt} while NCO binaries are from directory ${drc_nco}. Normally this script and the binaries are from the same executables directory. This INFO may be safely ignored for customized scripts and/or binaries that the user has intentionally split into different directories."
echo "HINT (All-users): Conflicting script and binary directories may result from 1) Hardcoding an NCO script and/or binary pathnames, 2) Incomplete NCO installations in one or more directories in the \$PATH environment variable, 3) (Re-)Installing or (re-)building NCO without issuing a \"hash -r\" command afterward to update the executable pathnames that the shell remembers from before."
echo "HINT (E3SM-only): In a Conda-based NCO environment, such as E3SM-Unified (which uses NCO in MPAS Analysis and E3SM-Diags), it is possible that some features may be unavailable because the upstream packages (e.g., UDUnits) were not properly linked by Conda. The ncclimo and ncremap scripts contain a mechanism to access the (presumably correctly linked) NCO binary executable and library paths in C. Zender's home directories on the major E3SM machines. Users may turn-on the machine-dependent, hard-coded path by invoking ncclimo and/or ncremap after altering their environment with: \"export NCO_PATH_OVERRIDE=Yes\"."
fi # drc_spt
# When running in a terminal window (not in an non-interactive batch queue)...
if [ -n "${TERM}" ]; then
# Set fonts for legibility
if [ -x /usr/bin/tput ] && tput setaf 1 &> /dev/null; then
fnt_bld=`tput bold` # Bold
fnt_nrm=`tput sgr0` # Normal
fnt_rvr=`tput smso` # Reverse
fnt_tlc=`tput sitm` # Italic
else
fnt_bld="\e[1m" # Bold
fnt_nrm="\e[0m" # Normal
fnt_rvr="\e[07m" # Reverse
fnt_tlc="\e[3m" # Italic
fi # !tput
fi # !TERM
# Pre-define enumerated types used in defaults
par_bck='background' # [sng] Parallelism: background
par_mpi='mpi' # [sng] Parallelism: MPI
par_srl='serial' # [sng] Parallelism: serial
# Defaults for command-line options and some derived variables
# Modify these defaults to save typing later
ann_sfx='01-01-00000' # [sng] Annual file suffix (MPAS, e.g., uses '01-01-00000')
area_nm='area' # [sng] Name of variable containing gridcell area
bch_pbs='No' # [sng] PBS batch (non-interactive) job
bch_slr='No' # [sng] SLURM batch (non-interactive) job
bnr_flg='No' # [sng] Binary method
caseid='' # [sng] Case ID
#caseid_xmp='20180129.DECKv1b_piControl.ne30_oEC.edison' # [sng] Case ID for v1 examples
caseid_xmp='v2.LR.historical_0101' # [sng] Case ID for v2 examples
caseid_xmp_eamxx='output.scream.decadal.monthlyAVG_ne30pg2.AVERAGE.nmonths_x1' # [sng] Case ID for EAMXX examples
cf_flg='Yes' # [sng] Produce CF climatology attribute?
clm_flg='Yes' # [sng] Generate climatology
clm_hfc_or_mth='No' # [sng] Generate monthly output climatology from monthly or HFC input
clm_md='mth' # [sng] Climatology mode ('ann', 'dly', 'hfc', 'hfs', or 'mth')
cmp_sng='' # [sng] Compression string
csn_flg='No' # [flg] Trigger season-processing code
csn_lst='mam,jja,son,djf' # [sng] Seasons to output (subset of mam,jja,son,djf,jfm,amj,jas,ond,fm,on)
d2f_flg='No' # [flg] Convert double-precision fields to single-precision
d2f_opt='--d2f' # [sng] Option string to convert double-precision fields to single-precision
dpt_fl='' # [sng] Depth file with refBottomDepth for MPAS ocean
dpt_opt='' # [sng] Option string to add depth coordinate to regridded MPAS files
dbg_lvl=0 # [nbr] Debugging level
dfl_lvl='' # [enm] Deflate level
drc_in='' # [sng] Input file directory
drc_in_xmp="${DATA}/ne30/raw" # [sng] Input file directory for examples
drc_in_mps="${DATA}/mpas/raw" # [sng] Input file directory for MPAS examples
drc_out="${drc_pwd}" # [sng] Output file directory
drc_out_xmp="${DATA}/ne30/clm" # [sng] Output file directory for examples
drc_out_mps="${DATA}/mpas/clm" # [sng] Output file directory for MPAS examples
drc_prv='' # [sng] Directory containing previous climatology to extend with current data
drc_rgr='' # [sng] Regridded file directory
drc_rgr_prv='' # [sng] Regridded file directory for previous climatology
drc_rgr_xmp="${DATA}/ne30/rgr" # [sng] Regrid file directory for examples
drc_rgr_xtn='' # [sng] Regridded file directory for for extended climatology
drc_xtn='' # [sng] Directory containing extended climatology
dpy=365 # [nbr] Days-per-year (noleap calendar)
fl_fmt='' # [enm] Output file format
fl_nbr=0 # [nbr] Number of files to split
fl_rmd=0 # [nbr] Number of files/months beyond an integral number of years
fml_nm='' # [sng] Family name (i.e., nickname, e.g., 'amip', 'control', 'experiment')
gaa_sng_std="--gaa climo_script=${spt_nm} --gaa climo_command=\"'${cmd_ln}'\" --gaa climo_hostname=${HOSTNAME} --gaa climo_version=${nco_vrs}" # [sng] Global attributes to add
hdr_pad='10000' # [B] Pad at end of header section
hrz_dmn='ncol' # [sng] Name of horizontal dimension(s) (e.g., 'lndgrid' and comma-separated if 2-D, e.g. 'lat,lon')
hst_nm='h0' # [sng] History volume (e.g., 'h0', 'h1', 'h')
inp_aut='No' # [sng] Input file list automatically generated
inp_glb='No' # [sng] Input file list from globbing directory
inp_psn='No' # [sng] Input file list from positional arguments
inp_std='No' # [sng] Input file list from stdin
job_nbr=12 # [nbr] Job simultaneity for parallelism
job_nbr_wrn=150 # [nbr] Maximum number of simultaneous jobs before WARNING is printed
lat_nm='lat' # [sng] Latitude variable name (e.g., 'lat')
lnk_flg='No' # [sng] Link E3SM-climo to AMWG-climo filenames
map_opt='' # [sng] Map-file options (e.g., '--map=map.nc')
mdl_nm='eam' # [sng] Model name string in history output (e.g., 'cam', 'cam2', 'cice', 'cism', 'clm', 'clm2', 'eam', 'elm', 'mali', 'mosart', 'mpascice', 'mpasseaice', 'mpassi', 'mpaso', 'mpasocean')
mdl_nm_usr='' # [sng] Model name string in history output (e.g., 'cam', 'cam2', 'cice', 'cism', 'clm', 'clm2', 'eam', 'elm', 'mali', 'mosart', 'mpascice', 'mpasseaice', 'mpassi', 'mpaso', 'mpasocean')
mdl_typ='cesm' # [sng] Model type ('cesm', 'mpas') (for regridding)
mem_mb=0 # [MB] Megabytes of RAM per srun job in Cori SLURM in MPI mode (0 indicates unlimited RAM)
mpi_flg='No' # [sng] Parallelize over nodes
mth_end='12' # [yr] End month
mth_flg='No' # [flg] Timeseries is not from Jan->Dec
mth_srt='1' # [yr] Start month
nco_opt='--no_tmp_fl' # [sng] NCO options (e.g., '-7 -D 1 -L 1')
ncr_flg='No' # [sng] Incremental method
nd_nbr=1 # [nbr] Number of nodes
nm_typ='cesm' # [sng] Name type ('cesm', 'mpas', 'yyyymm', 'yyyy-mm', 'yyyy-mm-01', 'yyyy-mm-01-00000') (for filenames)
no_ntv_tms='No' # [flg] Omit native-grid split timeseries
par_opt='' # [sng] Parallel options to shell
par_typ="${par_bck}" # [sng] Parallelism type for files
prc_typ='' # [sng] Procedure type (e.g., 'cam', 'cam2', 'cice', 'cism', 'clm', 'clm2', 'cpl', 'eam', 'eamxx', 'elm', 'mali', 'mpas', 'mpasa', 'mpasatmosphere', 'mpaso', 'mpasocean', 'mpascice', 'mpasseaice', 'mpassi')
qnt_prc='' # [nbr] Quantization precision
rgn_avg='No' # [sng] Regional/global average timeseries
rgn_stt='No' # [sng] Regional/global statistics
rgr_map='' # [sng] Regridding (horizontal) map
rgr_opt='' # [sng] Regridding options (e.g., '--rgr col_nm=lndgrid', '--rgr col_nm=nCells')
spl_opt='' # [sng] Splitter options (non-MPAS only) (e.g., '--no_cll_msr')
spl_rgr_opt='--no_stagger' # [sng] Splitter regridding options
sgs_frc='' # [sng] Sub-grid fraction variable
std_chk='Yes' # [sng] Check stdin for input file list
sum_scl='1.0f' # Factor to convert area, time units ([sr], [km2], [s]) to desired units (e.g., [m2], [yr]) in sums
thr_nbr=2 # [nbr] Thread number for regridder
tm_bnd_flg='Yes' # [sng] Time coordinate contains valid bounds variable
tms_flg='No' # [sng] Split (subset into timeseries) climatologies
tpd_out=1 # [nbr] Timesteps-per-day in output
#var_lst='FSNT,AODVIS' # [sng] Variables to process (empty means all)
var_lst='' # [sng] Variables to process (empty means all)
var_xtr='' # [sng] Extra variables for splitter timeseries
vrs_prn='No' # [sng] Print version information
vrt_out='' # [sng] Vertical grid file for output
vrt_opt='' # [sng] Vertical interpolation options (e.g., '--vrt_out=vrt_grd.nc')
vrt_xtr='' # [sng] Vertical extrapolation type (mss_val|nrs_ngh)
wnt_md='jfd' # [sng] Winter mode (contiguous: 'scd' | 'djf', discontiguous 'sdd' | 'jfd' as per above)
xcl_flg='No' # [sng] Exclude rather than extract variable list
xtn_flg='No' # [sng] Produce extended climatology
ypf_max=50 # [yr] Years-per-output-file
yr_end='2' # [yr] End year
yr_srt='1' # [yr] Start year
function fnc_usg_prn { # NB: dash supports fnc_nm (){} syntax, not function fnc_nm{} syntax
# Print usage
printf "${fnt_rvr}Basic usage:\n${fnt_nrm}${fnt_bld}${spt_nm} -c caseid -s yr_srt -e yr_end -i drc_in -o drc_out -r rgr_map${fnt_nrm} # Generate & regrid climatology\n"
printf "${fnt_bld}${spt_nm} --split -v var_lst -s yr_srt -e yr_end -o drc_out -r rgr_map in1.nc in2.nc ... inN.nc${fnt_nrm} # Split, reshape, & regrid timeseries\n"
printf "${fnt_bld}${spt_nm} --case=caseid --start=yr_srt --end=yr_end --input=drc_in --output=drc_out --map=rgr_map${fnt_nrm} # Long options (NB: '=' is REQUIRED)\n\n"
echo "Command-line options [long-option synonyms in ${fnt_tlc}italics${fnt_nrm}]:"
echo "${fnt_rvr}-3${fnt_nrm} Output file format CLASSIC (netCDF3 classic CDF1) [${fnt_tlc}fl_fmt, file_format=classic${fnt_nrm}]"
echo "${fnt_rvr}-4${fnt_nrm} Output file format NETCDF4 (netCDF4 extended HDF5) [${fnt_tlc}fl_fmt, file_format=netcdf4${fnt_nrm}]"
echo "${fnt_rvr}-5${fnt_nrm} Output file format 64BIT_DATA (netCDF3/PnetCDF CDF5) [${fnt_tlc}fl_fmt, file_format=64bit_data${fnt_nrm}]"
echo "${fnt_rvr}-6${fnt_nrm} Output file format 64BIT_OFFSET (netCDF3 64bit CDF2) [${fnt_tlc}fl_fmt, file_format=64bit_offset${fnt_nrm}]"
echo "${fnt_rvr}-7${fnt_nrm} Output file format NETCDF4_CLASSIC (netCDF4 classic HDF5) [${fnt_tlc}fl_fmt, file_format=netcdf4_classic${fnt_nrm}]"
echo "${fnt_rvr}-a${fnt_nrm} ${fnt_bld}wnt_md${fnt_nrm} Winter mode (default ${fnt_bld}${wnt_md}${fnt_nrm}) [${fnt_tlc}wnt_md, dcm_md, dec_md, december_mode, winter_mode${fnt_nrm}] [${fnt_tlc}scd,djf | sdd,jfd${fnt_nrm}]"
echo " ${fnt_bld}--amwg_lnk${fnt_nrm} Link E3SM-climo to AMWG-climo filenames [${fnt_tlc}amwg_links, AMWG_link${fnt_nrm}]"
echo " ${fnt_bld}--area_nm${fnt_nrm} Area-variable name (default ${fnt_bld}${area_nm}${fnt_nrm}) [${fnt_tlc}area_nm, area_name, area${fnt_nrm}]"
echo "${fnt_rvr}-C${fnt_nrm} ${fnt_bld}clm_md${fnt_nrm} Climatology mode (default ${fnt_bld}${clm_md}${fnt_nrm}) [${fnt_tlc}clm_md, climatology_mode, climo_mode${fnt_nrm}] [${fnt_tlc}ann | dly | hfc | hfs | mth${fnt_nrm}]"
echo "${fnt_rvr}-c${fnt_nrm} ${fnt_bld}caseid${fnt_nrm} Case ID string to generate input names or to name HFC output (default ${fnt_bld}${caseid}${fnt_nrm}) [${fnt_tlc}caseid, case_id, case${fnt_nrm}]"
echo " ${fnt_bld}--cmp_sng${fnt_nrm} Compression string (empty means none) (default ${fnt_bld}${cmp_sng}${fnt_nrm}) [${fnt_tlc}cmp, cmp_sng, compress, compression, cpr, cdc, codec ${fnt_nrm}]"
echo "${fnt_rvr}-d${fnt_nrm} ${fnt_bld}dbg_lvl${fnt_nrm} Debug level (default ${fnt_bld}${dbg_lvl}${fnt_nrm}) [${fnt_tlc}dbg_lvl, dbg, debug, debug_level${fnt_nrm}]"
echo " ${fnt_bld}--d2f${fnt_nrm} Convert double-precision fields to single-precision (default ${fnt_bld}${d2f_flg}${fnt_nrm}) [${fnt_tlc}d2f | d2s | dbl_flt | dbl_sgl | double_float${fnt_nrm}]"
echo " ${fnt_bld}--dpf${fnt_nrm} Days-per-file in every high-frequency splitter input file [${fnt_tlc}dpf, days_per_file${fnt_nrm}]"
echo " ${fnt_bld}--dpt_fl${fnt_nrm} Depth file with refBottomDepth for MPAS ocean (empty means none) (default ${fnt_bld}${dpt_fl}${fnt_nrm}) [${fnt_tlc}dpt_fl, mpas_fl, mpas_depth, depth_file${fnt_nrm}]"
echo "${fnt_rvr}-E${fnt_nrm} ${fnt_bld}yr_end${fnt_nrm} End year previous climo (empty means none) (default ${fnt_bld}${yr_end_prv}${fnt_nrm}) [${fnt_tlc}yr_end_prv, prv_yr_end, previous_end${fnt_nrm}]"
echo "${fnt_rvr}-e${fnt_nrm} ${fnt_bld}yr_end${fnt_nrm} End year (default ${fnt_bld}${yr_end}${fnt_nrm}) [${fnt_tlc}yr_end, end_yr, year_end, end_year, end${fnt_nrm}]"
echo "${fnt_rvr}-f${fnt_nrm} ${fnt_bld}fml_nm${fnt_nrm} Family name (nickname) (empty means none) (default ${fnt_bld}${fml_nm}${fnt_nrm}) [${fnt_tlc}fml_nm, fml, family_name${fnt_nrm}]"
echo " ${fnt_bld}--fl_fmt${fnt_nrm} File format (empty is inherited from input) (default ${fnt_bld}${fl_fmt}${fnt_nrm}) [${fnt_tlc}fl_fmt, fmt_out, file_format, format_out${fnt_nrm}]"
echo "${fnt_rvr}-h${fnt_nrm} ${fnt_bld}hst_nm${fnt_nrm} History volume name (default ${fnt_bld}${hst_nm}${fnt_nrm}) [${fnt_tlc}hst_nm, history_name, history${fnt_nrm}]"
echo " ${fnt_bld}--hrd_pth${fnt_nrm} Use CSZ's hard-coded paths on known machines (e.g., chrysalis, compy, perlmutter) NB: Must be first option! [${fnt_tlc}hrd_pth, hard_path, npo, nco_path_override${fnt_nrm}]"
echo "${fnt_rvr}-i${fnt_nrm} ${fnt_bld}drc_in${fnt_nrm} Input directory (default ${fnt_bld}${drc_in}${fnt_nrm}) [${fnt_tlc}drc_in, in_drc, dir_in, in_dir, input${fnt_nrm}]"
echo "${fnt_rvr}-j${fnt_nrm} ${fnt_bld}job_nbr${fnt_nrm} Job simultaneity for parallelism (default ${fnt_bld}${job_nbr}${fnt_nrm}) [${fnt_tlc}job_nbr, job_number, jobs${fnt_nrm}]"
echo "${fnt_rvr}-L${fnt_nrm} ${fnt_bld}dfl_lvl${fnt_nrm} Deflate level (empty is none) (default ${fnt_bld}${dfl_lvl}${fnt_nrm}) [${fnt_tlc}dfl_lvl, dfl, deflate${fnt_nrm}]"
echo "${fnt_rvr}-l${fnt_nrm} ${fnt_bld}lnk_flg${fnt_nrm} Link E3SM-climo to AMWG-climo filenames (default ${fnt_bld}${lnk_flg}${fnt_nrm}) [${fnt_tlc}lnk_flg, link_flag${fnt_nrm}]"
echo "${fnt_rvr}-m${fnt_nrm} ${fnt_bld}mdl_nm${fnt_nrm} Model name string in output filenames (default ${fnt_bld}${mdl_nm}${fnt_nrm}) [${fnt_tlc}mdl_nm, mdl, model_name, model${fnt_nrm}]"
echo " ${fnt_bld}--mth_end${fnt_nrm} End month (default ${fnt_bld}${mth_end}${fnt_nrm}) [${fnt_tlc}mth_end, end_mth, month_end, end_month${fnt_nrm}]"
echo " ${fnt_bld}--mth_srt${fnt_nrm} Start month (default ${fnt_bld}${mth_srt}${fnt_nrm}) [${fnt_tlc}mth_srt, start_mth, month_start, start_month${fnt_nrm}]"
echo "${fnt_rvr}-n${fnt_nrm} ${fnt_bld}nco_opt${fnt_nrm} NCO options (empty means none) (default ${fnt_bld}${nco_opt}${fnt_nrm}) [${fnt_tlc}nco_opt, nco, nco_options${fnt_nrm}]"
echo " ${fnt_bld}--no_lnk${fnt_nrm} Do not link E3SM-climo to AMWG-climo filenames [${fnt_tlc}no_links, no_amwg_links, no_amwg, no_AMWG_link${fnt_nrm}]"
echo " ${fnt_bld}--no_cll_msr${fnt_nrm} Omit cell_measures variables (e.g., 'area') [${fnt_tlc}no_area, no_cll_msr, no_cell_measures${fnt_nrm}]"
echo " ${fnt_bld}--no_frm_trm${fnt_nrm} Omit formula_terms variables (e.g., 'hyba', 'PS') [${fnt_tlc}no_frm_trm, no_frm, no_formula_terms${fnt_nrm}]"
echo " ${fnt_bld}--no_ntv_tms${fnt_nrm} Omit native-grid timeseries (splitter only) [${fnt_tlc}no_ntv_tms, no_ntv, no_native${fnt_nrm}]"
echo " ${fnt_bld}--no_stg_grd${fnt_nrm} Omit staggered grid variables ('slat, slon, w_stag') [${fnt_tlc}no_stg_grd, no_stg, no_stagger, no_staggered_grid${fnt_nrm}]"
echo " ${fnt_bld}--no_stdin${fnt_nrm} Do not check stdin for input file list [${fnt_tlc}no_stdin, no_inp_std, no_redirect, no_standard_input${fnt_nrm}]"
echo "${fnt_rvr}-O${fnt_nrm} ${fnt_bld}drc_rgr${fnt_nrm} Regridded directory (default ${fnt_bld}${drc_rgr}${fnt_nrm}) [${fnt_tlc}drc_rgr, rgr_drc, dir_regrid, regrid${fnt_nrm}]"
echo "${fnt_rvr}-o${fnt_nrm} ${fnt_bld}drc_out${fnt_nrm} Output directory (default ${fnt_bld}${drc_out}${fnt_nrm}) [${fnt_tlc}drc_out, out_drc, dir_out, out_dir, output${fnt_nrm}]"
echo "${fnt_rvr}-P${fnt_nrm} ${fnt_bld}prc_typ${fnt_nrm} Procedure type (empty means none) (default ${fnt_bld}${prc_typ}${fnt_nrm}) [${fnt_tlc}prc_typ, procedure${fnt_nrm}] (cpl|eam|eamxx|elm|mpaso|mpassi|cam|clm|pop|...)"
echo "${fnt_rvr}-p${fnt_nrm} ${fnt_bld}par_typ${fnt_nrm} Parallelism type for files (default ${fnt_bld}${par_typ}${fnt_nrm}) [${fnt_tlc}par_typ, par_md, parallel_type, parallel_mode, parallel${fnt_nrm}] [${fnt_tlc}serial | background | mpi${fnt_nrm}]"
echo " ${fnt_bld}--qnt_prc${fnt_nrm} Quantization precision (empty means none) (default ${fnt_bld}${qnt_prc}${fnt_nrm}) [${fnt_tlc}qnt, qnt_prc, ppc, ppc_prc, precision, quantize${fnt_nrm}]"
echo "${fnt_rvr}-R${fnt_nrm} ${fnt_bld}rgr_opt${fnt_nrm} Regrid options (empty means none) (default ${fnt_bld}${rgr_opt}${fnt_nrm}) [${fnt_tlc}rgr_opt, regrid_options${fnt_nrm}]"
echo "${fnt_rvr}-r${fnt_nrm} ${fnt_bld}rgr_map${fnt_nrm} Regrid map-file (horizontal) (empty means none) (default ${fnt_bld}${rgr_map}${fnt_nrm}) [${fnt_tlc}rgr_map, regrid_map, map, map_file, map_fl${fnt_nrm}]"
echo " ${fnt_bld}--rgn_avg${fnt_nrm} Regional/global average timeseries (default ${fnt_bld}${rgn_avg}${fnt_nrm}) [${fnt_tlc}rgn_avg, glb_avg, hms_avg, regional_average${fnt_nrm}]"
echo " ${fnt_bld}--rgn_stt${fnt_nrm} Regional/global statistical timeseries (default ${fnt_bld}${rgn_stt}${fnt_nrm}) [${fnt_tlc}rgn_stt, glb_stt, hms_stt, regional_statistic, global_statistic${fnt_nrm}] (avg|sum)"
echo "${fnt_rvr}-S${fnt_nrm} ${fnt_bld}yr_srt${fnt_nrm} Start year previous climo (empty means none) (default ${fnt_bld}${yr_srt_prv}${fnt_nrm}) [${fnt_tlc}yr_srt_prv, prv_yr_srt, previous_start${fnt_nrm}]"
echo "${fnt_rvr}-s${fnt_nrm} ${fnt_bld}yr_srt${fnt_nrm} Start year (default ${fnt_bld}${yr_srt}${fnt_nrm}) [${fnt_tlc}yr_srt, start_yr, year_start, start_year, start${fnt_nrm}]"
echo " ${fnt_bld}--seasons${fnt_nrm} Seasons to output ('none' means none) (default ${fnt_bld}${csn_lst}${fnt_nrm}, also available: jfm,amj,jas,ond,on,fm) [${fnt_tlc}csn_lst, csn, seasons${fnt_nrm}]"
echo " ${fnt_bld}--sgs_frc${fnt_nrm} Sub-grid fraction variable (empty means none) (default ${fnt_bld}${sgs_frc}${fnt_nrm}) [${fnt_tlc}sgs_frc, ice_frc, lnd_frc, ocn_frc, subgrid_fraction${fnt_nrm}]"
echo " ${fnt_bld}--split${fnt_nrm} Split input files into single-variable timeseries (default ${fnt_bld}${tms_flg}${fnt_nrm} [${fnt_tlc}splitter, tms_flg, timeseries${fnt_nrm}]"
echo " ${fnt_bld}--sum_scl${fnt_nrm} Scale factor to convert global sum timeseries to desired area units, timescale (empty means none) (default ${fnt_bld}${sum_scl}${fnt_nrm}) [${fnt_tlc}sum_scl, scl_fct, sum_scale, scale_factor${fnt_nrm}]"
echo "${fnt_rvr}-t${fnt_nrm} ${fnt_bld}thr_nbr${fnt_nrm} Thread number for regridder (default ${fnt_bld}${thr_nbr}${fnt_nrm}) [${fnt_tlc}thr_nbr, thr, thread_number, thread, threads${fnt_nrm}]"
echo " ${fnt_bld}--tpd_out${fnt_nrm} Timesteps-per-day in output (default ${fnt_bld}${tpd_out}${fnt_nrm}) [${fnt_tlc}tpd_out, tpd, timesteps_per_day${fnt_nrm}]"
echo " ${fnt_bld}--uio${fnt_nrm} Unbuffered I/O (NC_SHARE) for netCDF3 files [${fnt_tlc}uio, unbuffered, share${fnt_nrm}]"
echo "${fnt_rvr}-v${fnt_nrm} ${fnt_bld}var_lst${fnt_nrm} Variable list (empty means all) (default ${fnt_bld}${var_lst}${fnt_nrm}) [${fnt_tlc}var_lst, variable_list, var, vars, variable, variables${fnt_nrm}]"
echo " ${fnt_bld}--var_xtr${fnt_nrm} Extra variables for splitter timeseries (empty means none) (default ${fnt_bld}${var_xtr}${fnt_nrm}) [${fnt_tlc}var_xtr, var_extra, variables_extra, extra_variables${fnt_nrm}]"
echo " ${fnt_bld}--version${fnt_nrm} Version and configuration information [${fnt_tlc}version, vrs, config, configuration, cnf${fnt_nrm}]"
echo " ${fnt_bld}--vrt_out${fnt_nrm} Vertical grid file for output (empty means none) (default ${fnt_bld}${vrt_out}${fnt_nrm}) [${fnt_tlc}vrt_out, vrt_fl, vrt, vrt_crd, vrt_grd_out${fnt_nrm}]"
echo " ${fnt_bld}--vrt_xtr${fnt_nrm} Vertical extrapolation type (empty means none) (default ${fnt_bld}${vrt_xtr}${fnt_nrm}) [${fnt_tlc}vrt_xtr, xtr_mth, extrapolation_type, extrapolation_method${fnt_nrm}] (mss_val|nrs_ngh)"
echo "${fnt_rvr}-X${fnt_nrm} ${fnt_bld}drc_xtn${fnt_nrm} Extended climo directory (default ${fnt_bld}${drc_xtn}${fnt_nrm}) [${fnt_tlc}drc_xtn, xtn_drc, extended_dir, extended_climo, extended${fnt_nrm}]"
echo "${fnt_rvr}-x${fnt_nrm} ${fnt_bld}drc_prv${fnt_nrm} Previous climo directory (default ${fnt_bld}${drc_prv}${fnt_nrm}) [${fnt_tlc}drc_prv, prv_drc, previous_dir, previous_climo, previous${fnt_nrm}]"
echo " ${fnt_bld}--xcl_var${fnt_nrm} Exclude rather than extract var_lst [${fnt_tlc}xcl_var, xcl, exclude, exclude_variables${fnt_nrm}]"
echo "${fnt_rvr}-Y${fnt_nrm} ${fnt_bld}rgr_xtn${fnt_nrm} Regridded extended climo directory (default ${fnt_bld}${drc_rgr_xtn}${fnt_nrm}) [${fnt_tlc}drc_rgr_xtn, drc_xtn_rgr, regridded_extended, extended_regridded${fnt_nrm}]"
echo "${fnt_rvr}-y${fnt_nrm} ${fnt_bld}rgr_prv${fnt_nrm} Regridded previous climo directory (default ${fnt_bld}${drc_rgr_prv}${fnt_nrm}) [${fnt_tlc}drc_rgr_prv, drc_prv_rgr, regridded_previous, previous_regridded${fnt_nrm}]"
echo " ${fnt_bld}--ypf_max${fnt_nrm} Years-per-output-file maximum (default ${fnt_bld}${ypf_max}${fnt_nrm}) [${fnt_tlc}ypf_max, ypf, years, years_per_file${fnt_nrm}]"
printf "\n"
printf "${fnt_rvr}Examples:${fnt_nrm}\n${fnt_bld}${spt_nm} -P eam -c ${caseid_xmp} -s 2013 -e 2014 -i ${drc_in_xmp} -o ${drc_out_xmp} -r ~zender/data/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc ${fnt_nrm}# EAMv2 climo\n"
printf "${fnt_bld}${spt_nm} -P eamxx -v SW_flux_dn_at_model_bot,T_2m,wind_speed_10m -c ${caseid_xmp_eamxx} -f foo -s 1 -e 2 -i ${drc_in_xmp} -o ${drc_out_xmp} -r ~zender/data/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc ${fnt_nrm}# EAMXX climo\n"
printf "${fnt_bld}${spt_nm} -P elm -m clm2 -c control -s ${yr_srt} -e ${yr_end} -i ${drc_in_xmp} -o ${drc_out_xmp} -r ~zender/data/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc ${fnt_nrm}# ELMv2 climo\n"
printf "${fnt_bld}${spt_nm} -P mpaso -v timeMonthly_avg_activeTracers_temperature -s 1 -e 2 -i ${drc_in_mps} -o ${drc_out_mps} -r ~zender/data/maps/map_oEC60to30v3_to_cmip6_180x360_aave.20181001.nc ${fnt_nrm}# MPAS-Ocean v1 climo\n"
printf "${fnt_bld}${spt_nm} -P mpasseaice -v timeMonthly_avg_iceAreaCell -s ${yr_srt} -e ${yr_end} -i ${drc_in_mps} -o ${drc_out_mps} -r ~zender/data/maps/map_oRRS18to6v3_to_cmip6_720x1440_nco.20200601.nc ${fnt_nrm}# MPAS-Seaice v2 climo\n"
printf "${fnt_bld}cd output;ls *eam*19??-??*.nc | ${spt_nm} -P eam -v FSNT,TREFHT -s 1900 -e 1999 -o ${drc_out_xmp} -r ~zender/data/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc ${fnt_nrm}# EAM timeseries\n"
printf "${fnt_bld}ncclimo -c ${caseid_xmp} -S 41 -E 50 -x ${drc_rgr_xmp}/0041-0050 -s 51 -e 60 -i ${drc_rgr_xmp}/0051-0060 -X ${drc_rgr_xmp}/0041-0060 ${fnt_nrm}# Combine two EAM climos\n\n"
# 20210831: NB: Alternative access to interactive batch queues via, e.g., "salloc -A e3sm --constraint=knl --nodes=1 --time=00:30:00 --job-name=ncclimo" works well everywhere except Compy
printf "${fnt_rvr}Interactive batch queues:${fnt_nrm}\n"
printf "andes: srun -A CLI115 --nodes=1 --time=00:30:00 --job-name=ncclimo --pty bash\n"
printf "blues: srun -A condo --nodes=1 --partition=acme-small --time=00:30:00 --job-name=ncclimo --pty bash\n"
printf "cheye: qsub -I -A SCSG0002 -l select=1:ncpus=36:mpiprocs=1 -l walltime=00:30:00 -q regular -N ncclimo\n"
printf "chrys: srun -A e3sm --nodes=1 --partition=debug --time=00:30:00 --job-name=ncclimo --pty bash\n" # Login node
printf "chrys: srun --nodes=1 --time=30 --job-name=ncclimo --pty bash\n" # Compute node
printf "compy: srun -A e3sm --nodes=1 --time=00:30:00 --job-name=ncclimo --pty bash\n"
printf "coole: qsub -I -A ClimateEnergy_4 --nodecount=1 --time=00:30:00 --jobname=ncclimo\n"
printf "front: salloc -A CLI115 --nodes=1 --time=00:30:00 --job-name=ncclimo --pty bash\n"
printf "perlm: srun -A e3sm --constraint=cpu --nodes=1 --time=00:30:00 --qos=debug --job-name=ncclimo --pty bash\n"
printf "\nComplete documentation at http://nco.sf.net/nco.html#${spt_nm}\n\n"
exit 1
} # !fnc_usg_prn()
function bsh_mjr_vrs_lt_4 {
# Purpose: Return true if major version of Bash shell is < 4
# Usage: if bsh_mjr_vrs_lt_4 ; then ...; fi
local vrs
if [ -n ${BASH_VERSION} ]; then
let vrs=${BASH_VERSION%%.*} # NeR98 p. 100
if [ ${vrs} -lt 4 ] ; then # NeR98 p. 153
return 0; # 0 is true in shell-ese
fi # !vrs
fi # !BASH_VERSION
# By default, functions return exit status of last command executed
return 1; # 1 is false in shell-ese
} # !bsh_mjr_vrs_lt_4()
function trim_leading_zeros {
# Purpose: Trim leading zeros from string representing an integer
# Why, you ask? Because Bash treats zero-padded integers as octal!
# This is surprisingly hard to workaround
# My workaround is to remove leading zeros prior to arithmetic
# Usage: trim_leading zeros ${sng}
sng_trm=${1} # [sng] Trimmed string
# Use Bash 2.X pattern matching to remove up to three leading zeros, one at a time
sng_trm=${sng_trm##0} # NeR98 p. 99
sng_trm=${sng_trm##0}
sng_trm=${sng_trm##0}
# If all zeros removed, replace with single zero
if [ ${sng_trm} = '' ]; then
sng_trm='0'
fi # !sng_trm
} # !trim_leading_zeros()
# Check argument number and complain accordingly
arg_nbr=$#
#printf "\ndbg: Number of arguments: ${arg_nbr}"
if [ ${arg_nbr} -eq 0 ]; then
fnc_usg_prn
fi # !arg_nbr
# Parse command-line options:
# http://stackoverflow.com/questions/402377/using-getopts-in-bash-shell-script-to-get-long-and-short-command-line-options (see method by Adam Katz)
# http://tuxtweaks.com/2014/05/bash-getopts
while getopts :34567a:C:c:d:E:e:f:h:i:j:L:l:m:n:O:o:P:p:R:r:S:s:t:v:X:x:Y:y:-: OPT; do
case ${OPT} in
3) fl_fmt='3' ;; # File format
4) fl_fmt='4' ;; # File format
5) fl_fmt='5' ;; # File format
6) fl_fmt='6' ;; # File format
7) fl_fmt='7' ;; # File format
a) wnt_md_usr="${OPTARG}" ;; # Winter mode
C) clm_md_usr="${OPTARG}" ;; # Climatology mode
c) caseid="${OPTARG}" ;; # CASEID
d) dbg_lvl="${OPTARG}" ;; # Debugging level
E) yr_end_prv="${OPTARG}" ;; # End year previous
e) yr_end="${OPTARG}" ;; # End year
f) fml_nm_usr="${OPTARG}" ;; # Family name
h) hst_nm="${OPTARG}" ;; # History tape name
i) drc_in="${OPTARG}" ;; # Input directory
j) job_usr="${OPTARG}" ;; # Job simultaneity
L) dfl_lvl="${OPTARG}" ;; # Deflate level
l) lnk_flg="${OPTARG}" ;; # Link E3SM to AMWG name
m) mdl_nm_usr="${OPTARG}" ;; # Model name string
n) nco_opt="${OPTARG} ${nco_opt}" ;; # NCO options
o) drc_out_usr="${OPTARG}" ;; # Output directory
O) drc_rgr_usr="${OPTARG}" ;; # Regridded directory
P) prc_typ="${OPTARG}" ;; # Procedure type
p) par_typ="${OPTARG}" ;; # Parallelism type for files
R) rgr_opt_usr="${OPTARG}" ;; # Regridding options
r) rgr_map="${OPTARG}" ;; # Regridding map (horizontal)
S) yr_srt_prv="${OPTARG}" ;; # Start year previous
s) yr_srt="${OPTARG}" ;; # Start year
t) thr_usr="${OPTARG}" ;; # Thread number
v) var_lst="${OPTARG}" ;; # Variables
X) drc_xtn="${OPTARG}" ;; # Extended climo directory
x) drc_prv="${OPTARG}" ;; # Previous climo directory
Y) drc_rgr_xtn="${OPTARG}" ;; # Regridded extended climo directory
y) drc_rgr_prv="${OPTARG}" ;; # Regridded previous climo directory
z) ypf_max_usr="${OPTARG}" ;; # Years-per-output-file maximum
-) LONG_OPTARG="${OPTARG#*=}"
case ${OPTARG} in
# Hereafter ${OPTARG} is long argument key, and ${LONG_OPTARG}, if any, is long argument value
# Long options with no argument, no short option counterpart
# Long options with argument, no short option counterpart
# Long options with short counterparts, ordered by short option key
area_nm=?* | area_name=?* | area=?* ) area_nm="${LONG_OPTARG}" ;; # # Area-variable name
clm_md=?* | climatology_mode=?* | climo_mode=?* ) clm_md_usr="${LONG_OPTARG}" ;; # -C # Climatology mode
caseid=?* | case_id=?* | case=?* ) caseid="${LONG_OPTARG}" ;; # -c # CASEID
cmp=?* | cmp_sng=?* | compress=? | compression=?* | cpr=?* | cdc=?* | codec=?* ) cmp_sng="${LONG_OPTARG}" ;; # # Compression string
d2f | d2s | dbl_flt | dbl_sgl | double_float ) d2f_flg='Yes'; d2f_opt='--d2f' ;; # # Convert double-precision fields to single-precision
d2f=?* | d2s=?* | dbl_flt=?* | dbl_sgl=?* | double_float=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # D2F
wnt_md=?* | dcm_md=? | december_mode=?* | winter_mode=?* | dec_md=?* ) wnt_md="${LONG_OPTARG}" ;; # -a # Winter mode
dbg_lvl=?* | dbg=?* | debug=?* | debug_level=?* ) dbg_lvl="${LONG_OPTARG}" ;; # -d # Debugging level
dfl_lvl=?* | deflate=?* | dfl=?* ) dfl_lvl="${LONG_OPTARG}" ;; # -L # Deflate level
dpf=?* | days_per_file=?* ) dpf="${LONG_OPTARG}" ;; # # Days-per-file
dpt_fl=?* | mpas_fl=?* | mpas_file=?* | depth_file=?* ) dpt_opt="--dpt_fl=${LONG_OPTARG}" ;; # # Depth file with refBottomDepth for MPAS ocean
yr_end_prv=?* | prv_yr_end=?* | previous_end=?* ) yr_end_prv="${LONG_OPTARG}" ;; # -E # End year previous
yr_end=?* | end_yr=?* | year_end=?* | end_year=?* | end=?* ) yr_end="${LONG_OPTARG}" ;; # -e # End year
fml_nm=?* | fml=?* | family_name=?* | family=?* ) fml_nm_usr="${LONG_OPTARG}" ;; # -f # Family name
hst_nm=?* | history_name=?* | history=?* ) hst_nm="${LONG_OPTARG}" ;; # -h # History tape name
drc_in=?* | in_drc=?* | dir_in=?* | in_dir=?* | input=?* ) drc_in="${LONG_OPTARG}" ;; # -i # Input directory
fl_fmt=?* | fmt_out=?* | file_format=?* | format_out=?* ) fl_fmt="${LONG_OPTARG}" ;; # # Output file format
hrd_pth | hard_path | npo | nco_path_override | NCO_PATH_OVERRIDE ) hrd_pth='Yes' ;; # # Use hard-coded paths on known machines (intentional no-op because already handled prior to getopt())
hrd_pth=?* | hard_path=?* | npo=?* | nco_path_override=?* | NCO_PATH_OVERRIDE=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Use hard-coded paths on known machines
job_nbr=?* | job_number=?* | jobs=?* ) job_usr="${LONG_OPTARG}" ;; # -j # Job simultaneity
lnk_flg=?* | link_flag=?* ) lnk_flg="${LONG_OPTARG}" ;; # -l # Link E3SM to AMWG name
amwg_link | amwg_links | AMWG_link | AMWG_links ) lnk_flg='Yes' ;; # # Link E3SM to AMWG name
amwg_link=?* | amwg_links=?* | AMWG_link=?* | AMWG_links=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Link E3SM to AMWG name
no_amwg_link | no_amwg_links | no_amwg | no_AMWG_link | no_AMWG_links ) lnk_flg='No' ;; # # Link E3SM to AMWG name
no_amwg_link=?* | no_amwg_links=?* | no_amwg=?* | no_AMWG_link=?* | no_AMWG_links=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # -l # Link E3SM to AMWG name
mdl_nm=?* | mdl=?* | model_name=?* | model=?* ) mdl_nm_usr="${LONG_OPTARG}" ;; # -m # Model name string
mem_mb=?* | mmr_mb=?* | memory_mb=? | memory_megabytes=?* ) mem_mb="${LONG_OPTARG}" ;; # # Megabytes of RAM per srun job in Cori SLURM in MPI mode
mth_end=?* | end_mth=?* | month_end=?* | end_month=?* ) mth_end="${LONG_OPTARG}" ;; # # End month
mth_srt=?* | start_mth=?* | month_start=?* | start_month=?* ) mth_srt="${LONG_OPTARG}" ;; # # Start month
nco_opt=?* | nco=?* | nco_options=?* ) nco_opt="${LONG_OPTARG} ${nco_opt}" ;; # -n # NCO options
no_area | no_cll_msr | no_cell_measures ) no_cll_msr='Yes' ;; # # Omit cell_measures variables
no_area=?* | no_cell_msr=?* | no_cell_measures=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Omit cell_measures variables
no_frm_trm | no_frm | no_formula_terms ) no_frm_trm='Yes' ;; # # Omit formula_terms variables
no_frm_trm=?* | no_frm=?* | no_formula_terms=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Omit formula_terms variables
no_ntv_tms | no_ntv | no_native | no_native_timeseries | delete_native ) no_ntv_tms='Yes' ;; # # Omit native-grid split files
no_ntv_tms=?* | no_ntv=?* | no_native=?* | no_native_timeseries=?* | delete_native=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Omit native-grid split files
no_stg_grd | no_stg | no_stagger | no_staggered_grid ) no_stg_grd='Yes' ;; # # Omit staggered grid variables
no_stg_grd=?* | no_stg=?* | no_stagger=?* | no_staggered_grid ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Omit staggered grid variables
drc_out=?* | out_drc=?* | dir_out=?* | out_dir=?* | output=?* ) drc_out_usr="${LONG_OPTARG}" ;; # -o # Output directory
drc_rgr=?* | rgr_drc=?* | dir_regrid=?* | regrid_dir=?* | regrid=?* ) drc_rgr_usr="${LONG_OPTARG}" ;; # -O # Regridded directory
prc_typ=?* | procedure=?* ) prc_typ="${LONG_OPTARG}" ;; # -P # Procedure type
par_typ=?* | par_md=?* | parallel_type=?* | parallel_mode=?* | parallel=?* ) par_typ="${LONG_OPTARG}" ;; # -p # Parallelism type for files
qnt=?* | qnt_prc=?* | precision=?* | ppc=?* | ppc_prc=?* | quantize=?* ) qnt_prc="${LONG_OPTARG}" ;; # # Quantization precision
rgn_avg | glb_avg | hms_avg | regional_average ) rgn_avg='Yes'; rgn_stt='avg' ;; # # Regional/global average timeseries
rgn_avg=?* | glb_avg=?* | hms_avg=?* | regional_average=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Regional average timeseries
rgn_stt=?* | glb_stt=?* | hms_stt=?* | global_statistic=?* | regional_statistic=?* ) rgn_stt="${LONG_OPTARG}" ;; # # Regional/global statistical timeseries
rgr_opt=?* | regrid_options=?* ) rgr_opt_usr="${LONG_OPTARG}" ;; # -R # Regridding options
rgr_map=?* | regrid_map=?* | map=?* ) rgr_map="${LONG_OPTARG}" ;; # -r # Regridding map (horizontal)
yr_srt_prv=?* | prv_yr_srt=?* | previous_start=?* ) yr_srt_prv="${LONG_OPTARG}" ;; # -S # Start year previous
yr_srt=?* | start_yr=?* | year_start=?* | start_year=?* | start=?* ) yr_srt="${LONG_OPTARG}" ;; # -s # Start year
seasons=?* | csn_lst=?* | csn=?* | season=?* ) csn_lst="${LONG_OPTARG}" ;; # # Seasons to output
sgs_frc=?* | ice_frc=?* | lnd_frc=?* | ocn_frc=?* | subgrid_fraction=?* ) sgs_frc="${LONG_OPTARG}" ;; # # Sub-grid fraction variable
split | splitter | tms_flg | timeseries ) tms_flg='Yes' ;; # # Split input files into single-variable timeseries
split=?* | splitter=?* | tms_flg=?* | timeseries=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Split input files into single-variable timeseries
sum_scl=?* | scl_fct=?* | sum_scale=?* | scale_factor=?* ) sum_scl="${LONG_OPTARG}" ;; # # Factor to convert area, time units ([sr], [km2], [s]) to desired units (e.g., [m2], [yr]) in sums
stdin | inp_std | std_flg | redirect | standard_input ) printf "ERROR: ncclimo deprecated --stdin in version 5.0.4.\nHINT: Use --split to unambiguously invoke the splitter. This supercedes the old use for --stdin in ncclimo." ; exit 1 ;; # # Input file list from stdin
no_stdin | no_inp_std | no_redirect | no_standard_input ) std_chk='No' ;; # # Do not check stdin for input file list
no_stdin=?* | no_inp_std=?* | no_redirect=?* | no_standard_input=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Do not check stdin for input file list
thr_nbr=?* | thr=?* | thread_number=?* | thread=?* | threads=?* ) thr_usr="${LONG_OPTARG}" ;; # -t # Thread number
tpd_out=?* | tpd=?* | timesteps_per_day=?* ) tpd_usr="${LONG_OPTARG}" ;; # # Timesteps-per-day in output
uio | unbuffered | share ) uio_flg='Yes' ;; # # Unbuffered I/O (NC_SHARE) for netCDF3 files
uio=?* | unbuffered=?* | share=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Unbuffered I/O (NC_SHARE) for netCDF3 files
var_lst=?* | variable_list=?* | var=?* | vars=?* | variable=?* | variables=?* ) var_lst="${LONG_OPTARG}" ;; # -v # Variables
var_xtr=?* | var_extra=?* | variables_extra=?* | extra_variables=?* ) var_xtr=",${LONG_OPTARG}" ;; # # Extra variables for splitter timeseries (NB: initial comma is required/intentional"
version | vrs | config | configuration | cnf ) vrs_prn='Yes' ;; # # Print version information
version=?* | vrs=?* | config=?* | configuration=?* | cnf=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Print version information
vrt_out=?* | vrt_fl=?* | vrt=?* | vrt_crd=?* | vrt_grd_out=?* ) vrt_out="${LONG_OPTARG}" ;; # # Vertical grid file for output
vrt_xtr=?* | xtr_mth=?* | extrapolation_type=?* | extrapolation_method=?* ) vrt_xtr="${LONG_OPTARG}" ;; # # Vertical extrapolation type (mss_val|nrs_ngh)
drc_xtn=?* | xtn_drc=?* | extended_dir=?* | extended_climo=?* | extended=?* ) drc_xtn="${LONG_OPTARG}" ;; # -X # Extended climo directory
drc_prv=?* | prv_drc=?* | previous_dir=?* | previous_climo=?* | previous=?* ) drc_prv="${LONG_OPTARG}" ;; # -x # Previous climo directory
drc_rgr_xtn=?* | drc_xtn_rgr=?* | regridded_extended=?* | extended_regridded=?* ) drc_rgr_xtn="${LONG_OPTARG}" ;; # -Y # Regridded extended climo directory
drc_rgr_prv=?* | drc_prv_rgr=?* | regridded_previous=?* | previous_regridded=?* ) drc_rgr_prv="${LONG_OPTARG}" ;; # -y # Regridded previous climo directory
xcl_var | xcl | exclude | exclude_variables ) xcl_flg='Yes' ;; # # Exclude rather than extract variable list
xcl_var=?* | xcl=?* | exclude=?* | exclude_variables=?* ) echo "No argument allowed for --${OPTARG switch}" >&2; exit 1 ;; # # Exclude rather than extract variable list
ypf_max=?* | ypf=?* | years=?* | years_per_file=?* ) ypf_max_usr="${LONG_OPTARG}" ;; # -z # Years-per-output-file maximum
'' ) break ;; # "--" terminates argument processing
* ) printf "\nERROR: Unrecognized option ${fnt_bld}--${OPTARG}${fnt_nrm}\n" >&2; fnc_usg_prn ;;
esac ;; # !OPTARG
\?) # Unrecognized option
printf "\nERROR: Option ${fnt_bld}-${OPTARG}${fnt_nrm} not recognized\n" >&2
fnc_usg_prn ;;
esac # !OPT
done # !getopts
shift $((OPTIND-1)) # Advance one argument
psn_nbr=$#
if [ ${psn_nbr} -ge 1 ]; then
inp_psn='Yes'
# 20200430 Input files on command-line mean we need not check standard-input
std_chk='No'
fi # !psn_nbr
if [ ${vrs_prn} = 'Yes' ]; then
printf "${spt_nm}, the NCO climatology operator, version ${nco_vrs} \"${nco_sng}\"\n"
printf "Copyright (C) 2016--present Charlie Zender\n"
printf "This program is part of NCO, the netCDF Operators\n"
printf "NCO is free software and comes with a BIG FAT KISS and ABSOLUTELY NO WARRANTY\n"
printf "You may redistribute and/or modify NCO under the terms of the\n"
printf "3-Clause BSD License with exceptions described in the LICENSE file\n"
printf "BSD: https://opensource.org/licenses/BSD-3-Clause\n"
printf "LICENSE: https://github.com/nco/nco/tree/master/LICENSE\n"
printf "Config: ${spt_nm} script located in directory ${drc_spt}\n"
printf "Config: NCO binaries located in directory ${drc_nco}, linked to netCDF library version ${lbr_vrs}\n"
if [ "${hrd_pth_fnd}" = 'Yes' ]; then
printf "Config: Employ NCO machine-dependent hardcoded paths/modules for ${HOSTNAME}. (If desired, turn-off NCO hardcoded paths with \"export NCO_PATH_OVERRIDE=No\").\n"
else
printf "Config: No hardcoded machine-dependent path/module overrides. (If desired, turn-on NCO hardcoded paths at supported national labs with \"export NCO_PATH_OVERRIDE=Yes\").\n"
fi # !hrd_pth_fnd
exit 0
fi # !vrs_prn
# Detect input on pipe to stdin:
# http://stackoverflow.com/questions/2456750/detect-presence-of-stdin-contents-in-shell-script
# http://unix.stackexchange.com/questions/33049/check-if-pipe-is-empty-and-run-a-command-on-the-data-if-it-isnt
# 20170119 "if [ ! -t 0 ]" tests whether unit 0 (stdin) is connected to terminal, not whether pipe has data
# Non-interactive batch mode (e.g., qsub, sbatch) disconnects stdin from terminal and triggers false-positives with ! -t 0
# 20170123 "if [ -p foo ]" tests whether foo exists and is a pipe or named pipe
# Non-interactive batch mode (i.e., sbatch) behaves as desired for -p /dev/stdin on SLURM
# Non-interactive batch mode (e.g., qsub) always returns true for -p /dev/stdin on PBS, leads to FALSE POSITIVES!
# This is because PBS uses stdin to set the job name
# Hence -p /dev/stdin test works everywhere tested except PBS non-interactive batch environment
# Check stdin if user has not explicitly disallowed it with --no_stdin
if [ "${std_chk}" = 'Yes' ]; then
if [ -n "${PBS_ENVIRONMENT}" ]; then
if [ "${PBS_ENVIRONMENT}" = 'PBS_BATCH' ]; then
# PBS batch detection suggested by OLCF ticket CCS #338970 on 20170127
bch_pbs='Yes'
fi # !PBS_ENVIRONMENT
fi # !PBS
if [ -n "${SLURM_JOBID}" ] && [ -z "${SLURM_PTY_PORT}" ]; then
# SLURM batch detection suggested by NERSC ticket INC0096873 on 20170127
bch_slr='Yes'
fi # !SLURM
if [ ${bch_pbs} = 'Yes' ] || [ ${bch_slr} = 'Yes' ]; then
# Batch environment
if [ ${bch_pbs} = 'Yes' ]; then
if [ ! -p /dev/stdin ]; then
# PBS batch jobs cause -p to return true except for stdin redirection
# When -p returns true we do not know whether stdin pipe contains any input
# User must explicitly indicate use of stdin pipes with --stdin option
# Redirection in PBS batch jobs unambiguously causes -p to return false
inp_std='Yes'
fi # !stdin
fi # !bch_slr
if [ ${bch_slr} = 'Yes' ]; then
if [ -p /dev/stdin ]; then
# SLURM batch jobs cause -p to return true for stdin pipes
# When -p returns false we do not know whether output was redirected
# User must explicitly indicate use of redirection with --stdin option
# Stdin pipes in SLURM batch jobs unambiguously cause -p to return true
inp_std='Yes'
fi # !stdin
fi # !bch_slr
else # !bch
# Interactive environment
if [ -p /dev/stdin ] || [ ! -t 0 ]; then
# Interactive environments unambiguously cause -p to return true for stdin pipes
# Interactive environments unambiguously cause -t 0 to return false for stdin redirection
inp_std='Yes'
fi # !stdin
fi # !bch
if [ ${inp_std} = 'Yes' ] && [ ${inp_psn} = 'Yes' ]; then
echo "${spt_nm}: ERROR expecting input from both stdin and positional command-line arguments"
exit 1
fi # !inp_std
fi # !std_chk
# Determine mode first (this helps determine other defaults)
if [ -n "${yr_srt_prv}" ]; then
# Specifying only yr_srt_prv implies incremental method
# Specifying both yr_srt_prv and yr_end_prv implies binary method
xtn_flg='Yes'
if [ -n "${yr_end_prv}" ]; then
bnr_flg='Yes'
else # !yr_end_prv binary method
ncr_flg='Yes'
fi # !yr_end_prv binary method
fi # !yr_srt_prv extended climo
# Utilize user-specified model name, if any
if [ -n "${prc_typ}" ]; then
# 20201016: Supercede mdl_nm with prc_typ so both need not be explicitly invoked?
# EAM/ELM are difficult since history file name can be *cam*/*clm2* or *eam*/*elm*
# Instinct is to change mdl_nm so it only refers to name string in history file
# and must be provided only when that name string differs from default name string for prc_typ
prc_opt="-P ${prc_typ}"
mdl_nm="${prc_typ}"
# Exceptions to rule that mdl_nm == prc_typ
[[ "${prc_typ}" = 'clm' ]] && mdl_nm='clm2'
[[ "${prc_typ}" = 'sgs' ]] && mdl_nm='elm'
fi # !prc_typ
if [ -n "${mdl_nm_usr}" ]; then
mdl_nm="${mdl_nm_usr}"
fi # !mdl_nm_usr
if [ -n "${wnt_md_usr}" ]; then
if [ "${wnt_md_usr}" = 'djf' ] || [ "${wnt_md_usr}" = 'DJF' ] || [ "${wnt_md_usr}" = 'scd' ] || [ "${wnt_md_usr}" = 'seasonally_contiguous_december' ]; then
wnt_md='djf'
elif [ "${wnt_md_usr}" = 'jfd' ] || [ "${wnt_md_usr}" = 'JFD' ] || [ "${wnt_md_usr}" = 'sdd' ] || [ "${wnt_md_usr}" = 'seasonally_discontiguous_december' ]; then
wnt_md='jfd'
fi # !wnt_md_usr
fi # !wnt_md_usr
if [ -n "${clm_md_usr}" ]; then
# Climo mode must be explicitly selected with --clm_md when climo input files are from stdin or positional
if [ "${clm_md_usr}" = 'ann' ] || [ "${clm_md_usr}" = 'annual' ] || [ "${clm_md_usr}" = 'yearly' ] || [ "${clm_md_usr}" = 'year' ]; then
clm_md_usr='ann'
fi # !clm_md_usr
if [ "${clm_md_usr}" = 'dly' ] || [ "${clm_md_usr}" = 'daily' ] || [ "${clm_md_usr}" = 'doy' ] || [ "${clm_md_usr}" = 'day' ]; then
clm_md_usr='dly'
fi # !clm_md_usr
if [ "${clm_md_usr}" = 'hfc' ] || [ "${clm_md_usr}" = 'high_frequency_climo' ] || [ "${clm_md_usr}" = 'hgh_frq_clm' ]; then
clm_md_usr='hfc'
fi # !clm_md_usr
if [ "${clm_md_usr}" = 'hfs' ] || [ "${clm_md_usr}" = 'high_frequency_splitter' ] || [ "${clm_md_usr}" = 'hgh_frq_spl' ]; then
clm_md_usr='hfs'
fi # !clm_md_usr
if [ "${clm_md_usr}" = 'mth' ] || [ "${clm_md_usr}" = 'month' ] || [ "${clm_md_usr}" = 'monthly' ]; then
clm_md_usr='mth'
fi # !clm_md_usr
clm_md="${clm_md_usr}"
fi # !clm_md_usr
if [ "${clm_md}" = 'hfc' ] || [ "${clm_md}" = 'mth' ]; then
clm_hfc_or_mth='Yes'
fi # !clm_md
if [ "${clm_md}" = 'dly' ] || [ "${clm_md}" = 'hfc' ]; then
unset dpm # Days per month
declare -a dpm
dpm=(0 31 28 31 30 31 30 31 31 30 31 30 31) # noleap 365-day calendar, 1-based indexing
fi # !clm_md
if [ -z "${drc_in}" ]; then
drc_in="${drc_pwd}"
else # !drc_in
if [ ! -d "${drc_in}" ]; then
echo "${spt_nm}: ERROR specified input directory \"${drc_in}\" does not exist"
exit 1
fi # !drc_in
drc_in_usr_flg='Yes'
fi # !drc_in
# Derived variables
if [ -n "${drc_out_usr}" ]; then
# Fancy %/ syntax removes trailing slash (e.g., from $TMPDIR)
drc_out="${drc_out_usr%/}"
fi # !drc_out_usr
if [ -n "${drc_rgr_usr}" ]; then
if [ -z "${rgr_map}" ]; then
printf "${spt_nm}: ERROR specified directory for regridded files without supplying map-file name for regridding\n"
printf "${spt_nm}: HINT Provide a map-file name with -m map.nc or --map=map.nc or remove the option (-O or --drc_rgr) that sets the directory for regridded files\n"
exit 1
fi # !err
drc_rgr="${drc_rgr_usr%/}"
else
drc_rgr="${drc_out%/}"
fi # !drc_rgr_usr
if [ -n "${drc_prv}" ]; then
drc_prv="${drc_prv%/}"
else
if [ "${bnr_flg}" = 'Yes' ]; then
drc_prv="${drc_in}"
fi # !bnr_flg
if [ "${ncr_flg}" = 'Yes' ]; then
drc_prv="${drc_out}"
fi # !ncr_flg
fi # !drc_prv
if [ -n "${drc_xtn}" ]; then
drc_xtn="${drc_xtn%/}"
else
drc_xtn="${drc_prv}"
fi # !drc_xtn
# Doubly-derived variables
if [ -n "${drc_rgr_prv}" ]; then
drc_rgr_prv="${drc_rgr_prv%/}"
else
drc_rgr_prv="${drc_prv%/}"
fi # !drc_rgr_prv
if [ -n "${drc_rgr_xtn}" ]; then
drc_rgr_xtn="${drc_rgr_xtn%/}"
else
drc_rgr_xtn="${drc_xtn%/}"
fi # !drc_rgr_xtn
# Create output directories
if [ -n "${drc_out}" ] && [ ! -d "${drc_out}" ]; then
chr_fst=${drc_out:0:1}
if [ "${chr_fst}" = '-' ]; then
echo "${spt_nm}: ERROR Attempting to mkdir user-specified output directory \"${drc_out}\" will fail because directory name begins with '-' which is an option indicator"
echo "${spt_nm}: HINT Specify output directory name that does not begin with '-'"
exit 1
fi # !chr_fst
cmd_mkd="mkdir -p ${drc_out}"
eval ${cmd_mkd}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR Failed to create output directory. Debug this:\n${cmd_mkd}\n"
printf "${spt_nm}: HINT Creating a directory requires proper write permissions\n"
exit 1
fi # !err
fi # !drc_out
if [ -n "${drc_rgr}" ] && [ ! -d "${drc_rgr}" ]; then
chr_fst=${drc_rgr:0:1}
if [ "${chr_fst}" = '-' ]; then
echo "${spt_nm}: ERROR Attempting to mkdir user-specified regrid directory \"${drc_rgr}\" will fail because directory name begins with '-' which is an option indicator"
echo "${spt_nm}: HINT Specify regrid directory name that does not begin with '-'"
exit 1
fi # !chr_fst
cmd_mkd="mkdir -p ${drc_rgr}"
eval ${cmd_mkd}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR Attempt to create regrid directory. Debug this:\n${cmd_mkd}\n"
printf "${spt_nm}: HINT Creating a directory requires proper write permissions\n"
exit 1
fi # !err
fi # !drc_rgr
# Define calendar
# Determine first full year
trim_leading_zeros ${yr_srt}
yr_srt_rth=${sng_trm}
yyyy_srt=`printf "%04d" ${yr_srt_rth}`
let yr_srtm1=${yr_srt_rth}-1
trim_leading_zeros ${yr_end}
yr_end_rth=${sng_trm}
yyyy_end=`printf "%04d" ${yr_end_rth}`
let yr_endm1=${yr_end_rth}-1
let yr_endp1=${yr_end_rth}+1
let yr_nbr=${yr_end_rth}-${yr_srt_rth}+1
if [ ${mth_srt} -eq '0' ]; then
echo "${spt_nm}: ERROR User-defined start month is ${mth_srt}. Please use 1-based (not 0-based) indexing for months, where [Jan..Dec] = [1..12]"
exit 1
fi # !mth_srt
trim_leading_zeros ${mth_srt}
mth_srt_rth=${sng_trm}
mm_srt=`printf "%02d" ${mth_srt_rth}`
trim_leading_zeros ${mth_end}
mth_end_rth=${sng_trm}
mm_end=`printf "%02d" ${mth_end_rth}`
let mth_nbr=${yr_end_rth}*12+${mth_end_rth}-${yr_srt_rth}*12-${mth_srt_rth}+1
let mth_srtm1=${mth_srt_rth}-1
let mth_endp1=${mth_end_rth}+1
if [ ${mth_srtm1} -eq 0 ]; then
let mth_srtm1=12
mm_srtm1='12'
else # !mth_srtm1
mm_srtm1=`printf "%02d" ${mth_srtm1}`
fi # !mth_srtm1
if [ ${mth_endp1} -eq 13 ]; then
let mth_endp1=1
mm_endp1='01'
else # !mth_endp1
mm_endp1=`printf "%02d" ${mth_endp1}`
fi # !mth_endp1
if [ "${mm_srt}" != '01' ] || [ "${mm_end}" != '12' ]; then
mth_flg='Yes'
fi # !mm_srt
# Define actions based on standard options
if [ -n "${caseid}" ]; then
out_nm=${caseid}
fi # !caseid
if [ "${caseid}" = 'hist' ] || [ "${mdl_nm}" = 'mali' ] || [ "${mdl_nm}" = 'mpas' ] || [ "${mdl_nm}" = 'mpaso' ] || [ "${mdl_nm}" = 'mpas-ocean' ] || [ "${mdl_nm}" = 'mpascice' ] || [ "${mdl_nm}" = 'mpasseaice' ] || [ "${mdl_nm}" = 'mpas-seaice' ] || [ "${mdl_nm}" = 'mpassi' ] || [ "${prc_typ}" = 'mali' ] || [ "${prc_typ}" = 'mpas' ] || [ "${prc_typ}" = 'mpasa' ] || [ "${prc_typ}" = 'mpasatmosphere' ] || [ "${prc_typ}" = 'mpaso' ] || [ "${prc_typ}" = 'mpas-ocean' ] || [ "${prc_typ}" = 'mpascice' ] || [ "${prc_typ}" = 'mpasseaice' ] || [ "${prc_typ}" = 'mpas-seaice' ] || [ "${prc_typ}" = 'mpassi' ]; then
mdl_typ='mpas'
nm_typ='mpas'
out_nm="${mdl_nm}"
hst_nm='hist'
fi # !caseid
if [ "${prc_typ}" = 'mpascice' ] || [ "${prc_typ}" = 'mpasseaice' ] || [ "${prc_typ}" = 'mpassi' ]; then
prc_mpas='Yes'
prc_mpasseaice='Yes'
fi # !prc_typ
if [ "${mdl_typ}" = 'mpas' ] && [ -z "${prc_opt}" ]; then
prc_typ='mpas'
prc_opt='-P mpas'
fi # !mdl_typ && prc_opt
# http://stackoverflow.com/questions/965053/extract-filename-and-extension-in-bash
# http://stackoverflow.com/questions/17420994/bash-regex-match-string
# NB: Order and 'elif' ensure name matches only fullest rx
if [[ "${caseid}" =~ ^(.*)([0-9][0-9][0-9][0-9][01][0-9].nc.?)$ ]]; then
nm_typ='yyyymm'
elif [[ "${caseid}" =~ ^(.*)([0-9][0-9][0-9][0-9]-[01][0-9]-01.nc.?)$ ]]; then
nm_typ='yyyy-mm-01'
elif [[ "${caseid}" =~ ^(.*)([0-9][0-9][0-9][0-9]-[01][0-9]-01-00000.nc.?)$ ]]; then
nm_typ='yyyy-mm-01-00000'
elif [[ "${caseid}" =~ ^(.*)([0-9][0-9][0-9][0-9]-[01][0-9].nc.?)$ ]]; then
nm_typ='yyyy-mm'
fi # !caseid
if [ "${nm_typ}" = 'yyyymm' ] || [ "${nm_typ}" = 'yyyy-mm' ] || [ "${nm_typ}" = 'yyyy-mm-01' ] || [ "${nm_typ}" = 'yyyy-mm-01-00000' ]; then
bs_nm="${BASH_REMATCH[1]}"
bs_nm="$(basename ${bs_nm})"
bs_nm="${bs_nm%.*}"
[[ ${prc_typ} != 'eamxx' ]] && bs_nm="${bs_nm%_*}"
out_nm=${bs_nm}
# https://stackoverflow.com/questions/12426659/how-extract-last-part-of-string-in-bash
bs_sfx="${caseid##*.}"
fi # !nm_typ
if [ -n "${fml_nm_usr}" ]; then
fml_nm="${fml_nm_usr}"
out_nm="${fml_nm}"
fi # !fml_nm
# Determine whether to compute climos or to split
# Original behavior:
# Until 20211109, stdin and positional always indicated splitter even when ypf was not set
# New ability to do monthly climos with filenames provided by stdin, positional, or globbing necessitated modifications to original algorithm to determine climo vs. splitter
# As of 20211129, default behavior is to compute climo
# One of three methods may be used to invoke the splitter:
# 1) Use --split flag. 2) Select clm_md=hfs. 3) Set ypf_max.
# NB: clm_md='mth' indicates only the averaging period of a single input file
# clm_md='mth' is true for both monthly splitter and monthly climo :(
# Rest of code relies on (mutually exclusive) clm_flg to compute climo and tms_flg to invoke splitter
if [ "${tms_flg}" = 'Yes' ] || [ "${clm_md}" = 'hfs' ] || [ -n "${ypf_max_usr}" ]; then
tms_flg='Yes'
clm_flg='No'
if [ -n "${ypf_max_usr}" ]; then
ypf_max=${ypf_max_usr}
fi # !ypf_max_usr
wnt_md='jfd'
fi # !tms_flg
# Determine input method, if not already known
if [ ${inp_psn} = 'No' ] && [ ${inp_std} = 'No' ] && [ "${drc_in_usr_flg}" = 'Yes' ]; then
# 20220129 Determine whether to construct (use inp_aut) or glob files
# Daily (or finer) resolution filenames are never constructible
# Monthly input filenames are constructable with caseid
# Assume user wishes to glob unless both are provided caseid for monthly files
# 20211207 Specifying mpas name type (like MPAS-Analysis does) also implies inp_aut
if [[ ( ${clm_md} = 'mth' && -n "${caseid}" ) || ${nm_typ} = 'mpas' ]]; then
inp_aut='Yes'
else # !mth
inp_glb='Yes'
fi # !caseid
fi # !inp_*
if [ ${clm_flg} = 'Yes' ] && [ ${clm_md} = 'ann' ]; then
inp_aut='Yes'
inp_glb='No'
inp_std='No' # fxm: 20170123 hack for false positives in non-interactive batch mode on PBS
fi # !clm_flg
if [ ${clm_flg} = 'Yes' ] && [ ${clm_md} = 'dly' ] && [ ${inp_psn} = 'No' ] && [ ${inp_std} = 'No' ] && [ "${drc_in_usr_flg}" = 'Yes' ]; then
inp_aut='No'
inp_glb='Yes'
fi # !clm_flg, !dly
# One of these input file options must be true, or else...
if [ ${inp_aut} = 'No' ] && [ ${inp_glb} = 'No' ] && [ ${inp_psn} = 'No' ] && [ ${inp_std} = 'No' ]; then
echo "${spt_nm}: ERROR Specify input file(s) by providing option \"-i \$drc_in\" with (for automatic filename construction) or without (for globbing entire directory) \"-c \$caseid\", or provide names via positional argument(s) or stdin"
if [ ${bch_pbs} = 'Yes' ]; then
echo "${spt_nm}: HINT PBS batch job environment detected, pipe to stdin not allowed, try positional arguments instead"
else # !bch_pbs
echo "${spt_nm}: HINT Provide a list of lengthy or complex filenames as a pipe to stdin with, e.g., 'ls *.nc | ${spt_nm}'"
fi # !bch_pbs
exit 1
fi # !tms_flg
if [ "${inp_aut}" != 'Yes' ] && [ -z "${mdl_nm_usr}" ]; then
mdl_nm='nil' # Unset default model name for stdin pipe and positional arguments, otherwise default mdl_nm could be used to add model-specific variables to var_xtr
fi # !inp_aut !mdl_nm_usr
# clm_flg and tms_flg and all input file pre-requisite parameters are now set. Ready to gather input filenames.
# Obtain filenames from automatic generation, directory glob, positional arguments, or stdin pipe
#printf "dbg: inp_aut = ${inp_aut}, inp_glb = ${inp_glb}, inp_psn = ${inp_psn}, inp_std = ${inp_std}\n"
# End input mode determination block
if [ "${clm_flg}" = 'Yes' ] && [ -z "${out_nm}" ] && [ -z "${fml_nm}" ]; then
printf "${spt_nm}: ERROR ${spt_nm} User requested climatology generation yet invoked neither \$caseid nor \$fml_nm options nor specified a supported automatic name-generation option (e.g., --mdl=mpaso for MPAS) so ${spt_nm} does not know what to name the output files\n"
printf "${spt_nm}: HINT Use \"--caseid=name\" or \"--fml_nm=name\" to provide a name for the output files. If the input files are provided via stdin, positional arguments, or globbing then the name string will only be used for output names and need not be present in any of the input filenames\n"
fi # !out_nm
if [ "${clm_md}" != 'ann' ] && [ "${clm_md}" != 'dly' ] && [ "${clm_md}" != 'hfc' ] && [ "${clm_md}" != 'hfs' ] && [ "${clm_md}" != 'mth' ]; then
echo "${spt_nm}: ERROR User-defined climatology mode is ${clm_md}. Valid options are 'ann', 'dly', hfc', 'hfs', and 'mth' (default)"
exit 1
fi # !clm_md
if [ "${clm_md}" = 'ann' ]; then
clm_nbr=1
wnt_md='jfd'
elif [ "${clm_md}" = 'dly' ]; then
clm_nbr=${dpy}
wnt_md='jfd'
elif [ "${clm_hfc_or_mth}" = 'Yes' ]; then
clm_nbr=12
csn_flg='Yes'
if [ ${csn_lst} = 'none' ]; then
csn_nbr=0
csn_flg='No'
fi # !csn_lst
fi # !clm_md
if [ -n "${gaa_sng_std}" ]; then
if [ "${yr_nbr}" -gt 1 ]; then
yrs_avg_sng="${yr_srt}-${yr_end}"
else
yrs_avg_sng="${yr_srt}"
fi # !yr_nbr
# NB: E3SM diagnostics (e3sm_diagnostics) relies on this attribute
if [ "${tms_flg}" != 'Yes' ]; then
gaa_sng="${gaa_sng_std} --gaa yrs_averaged=${yrs_avg_sng}"
fi # !tms_flg
else
echo "${spt_nm}: INFO User-defined global attributes string is empty. Diagnostics routines (such as E3SM Diagnostics) that depend on these attributes may not work."
fi # !gaa_sng
if [ -n "${job_usr}" ]; then
job_nbr="${job_usr}"
fi # !job_usr
if [ -n "${fl_fmt}" ]; then
if [ "${fl_fmt}" = '3' ] || [ "${fl_fmt}" = 'classic' ] || [ "${fl_fmt}" = 'netcdf3' ]; then
nco_fl_fmt='--fl_fmt=classic'
elif [ "${fl_fmt}" = '4' ] || [ "${fl_fmt}" = 'netcdf4' ] || [ "${fl_fmt}" = 'hdf5' ]; then
nco_fl_fmt='--fl_fmt=netcdf4'
elif [ "${fl_fmt}" = '5' ] || [ "${fl_fmt}" = '64bit_data' ] || [ "${fl_fmt}" = 'cdf5' ]; then
nco_fl_fmt='--fl_fmt=64bit_data'
elif [ "${fl_fmt}" = '6' ] || [ "${fl_fmt}" = '64bit_offset' ] || [ "${fl_fmt}" = '64' ]; then
nco_fl_fmt='--fl_fmt=64bit_offset'
elif [ "${fl_fmt}" = '7' ] || [ "${fl_fmt}" = 'netcdf4_classic' ]; then
nco_fl_fmt='--fl_fmt=netcdf4_classic'
else
echo "${spt_nm}: ERROR User-supplied file-format specifier fl_fmt='${fl_fmt}' is invalid"
echo "${spt_nm}: HINT Valid format-specifiers include '3', '4', '5', '6', and '7'"
exit 1
fi # !fl_fmt
nco_opt="${nco_fl_fmt} ${nco_opt}"
fi # !fl_fmt
if [ ${dbg_lvl} -ge 2 ]; then
nco_opt="--dbg_lvl=${dbg_lvl} ${nco_opt}"
fi # !dbg_lvl
# 20240617 var_lst for timeseries mode must be and is processed separately in timeseries block
if [ -n "${var_lst}" ] && [ "${tms_flg}" != 'Yes' ]; then
if [ "${xcl_flg}" = 'Yes' ]; then
nco_opt="${nco_opt} -x"
fi # !xcl_flg
nco_opt="${nco_opt} -v ${var_lst}"
fi # !var_lst
if [ -n "${cmp_sng}" ]; then
# 20230623: Insert backslashes to protect pipe characters from shell
nco_opt="${nco_opt} --cmp=${cmp_sng//\|/\\|}"
fi # !cmp_sng
if [ -n "${dfl_lvl}" ]; then
nco_opt="${nco_opt} --dfl_lvl=${dfl_lvl}"
fi # !dfl_lvl
if [ -n "${qnt_prc}" ]; then
nco_opt="${nco_opt} --qnt default=${qnt_prc}"
fi # !qnt_prc
if [ -n "${hdr_pad}" ]; then
nco_opt="${nco_opt} --hdr_pad=${hdr_pad}"
fi # !hdr_pad
if [ -n "${uio_flg}" ]; then
nco_opt="${nco_opt} --unbuffered_io"
fi # !uio_flg
if [ "${no_cll_msr}" = 'Yes' ]; then
spl_opt="${spl_opt} --no_cll_msr"
fi # !no_cll_msr
if [ "${no_frm_trm}" = 'Yes' ]; then
spl_opt="${spl_opt} --no_frm_trm"
fi # !no_frm_trm
if [ "${no_stg_grd}" = 'Yes' ]; then
spl_rgr_opt=''
fi # !no_stg_grd
if [ "${par_typ}" = ${par_bck} ] || [[ "${par_typ}" =~ [bB]ck ]] || [[ "${par_typ}" =~ [bB]ackground ]]; then
par_typ=${par_bck}
par_opt=' &'
elif [ "${par_typ}" = ${par_mpi} ] || [[ "${par_typ}" =~ (mpi|MPI) ]]; then
par_typ=${par_mpi}
par_opt=' &'
mpi_flg='Yes'
elif [ "${par_typ}" = ${par_srl} ] || [ "${par_typ}" = 'srl' ] || [[ "${par_typ}" =~ [sS]erial ]] || [[ "${par_typ}" =~ [nN]il ]] || [[ "${par_typ}" =~ [nN]one ]]; then
par_typ=${par_srl}
else
echo "ERROR: Invalid -p par_typ option = ${par_typ}"
echo "HINT: Valid par_typ arguments for file-level parallelism include '${par_bck}' (or 'bck'), '${par_mpi}' (or 'MPI'), and '${par_srl}' (or 'srl' or 'nil' or 'none'). For background parallelism, select '${par_bck}' which causes ${spt_nm} to spawn parallel processes as background tasks on a single node. For MPI parallelism, select '${par_mpi}' which causes ${spt_nm} to spawn parallel processes on across available cluster nodes. For no parallelism (aka serial mode), select '${par_srl}', which causes ${spt_nm} to spawn all processes serially on a single compute node."
exit 1
fi # !par_typ
if [ -n "${rgr_opt_usr}" ]; then
rgr_opt="${rgr_opt_usr}"
rmp_opt="-R \'${rgr_opt_usr}\'"
fi # !rgr_opt_usr
if [ -n "${rgr_map}" ]; then
if [ ! -f "${rgr_map}" ] && [ ! -L "${rgr_map}" ]; then
echo "${spt_nm}: ERROR Unable to find specified regrid map ${rgr_map}"
echo "${spt_nm}: HINT Supply the full path-name for the regridding map"
exit 1
fi # ! -f
if [ "${mdl_nm}" = 'clm2' ] || [ "${mdl_nm}" = 'elm' ] && [ -z "${sgs_frc}" ]; then
printf "${spt_nm}: WARNING ${spt_nm} CLM/ELM output typically uses sub-gridscale (SGS) fractions that can only be properly renormalized if the SGS variable option to ncclimo is invoked, but that option was not invoked\n"
printf "${spt_nm}: HINT Provide ${spt_nm} with the SGS fraction variable using the \"--sgs_frc=\${sgs_frc_nm}\" option, otherwise the regridded data will likely not be properly or conservatively regridded\n"
fi # !mdl_nm
map_opt="--map=${rgr_map}"
rgr_opt="${rgr_opt} ${map_opt}"
rmp_opt="${rmp_opt} ${map_opt}"
else # !rgr_map
if [ -n "${dpt_opt}" ]; then
echo "${spt_nm}: ERROR Cannot add depth coordinate unless horizontal regridding is invoked"
echo "${spt_nm}: HINT Specify a regridding map or run add_depth.py on these files yourself. To request that add_depth.py work directly in ncclimo, contact Charlie."
exit 1
fi # !dpt_opt
fi # !rgr_map
if [ "${tms_flg}" = 'Yes' ]; then
# 20190419: Splitter typically writes 1--3 multi-dimensional fields into each output file
# OpenMP incurs significant RAM overhead that may not be worthwhile
# New default is 1 regridding thread for splitter
# Users can override this splitter-only default by explicitly requesting multiple threads
thr_nbr=1 # [nbr] Thread number for regridder
fi # !tms_flg
if [ -n "${sgs_frc}" ]; then
sgs_opt="--rgr sgs_frc_nm=${sgs_frc}"
rgr_opt="${rgr_opt} ${sgs_opt}"
rmp_opt="--sgs_frc=${sgs_frc} ${rmp_opt}"
fi # !sgs_frc
if [ -n "${thr_usr}" ]; then
thr_nbr="${thr_usr}"
fi # !thr_usr
if [ -n "${tpd_usr}" ]; then
tpd_out="${tpd_usr}"
fi # !tpd_usr
if [ -n "${vrt_out}" ]; then
if [ ! -f "${vrt_out}" ] && [ ! -L "${vrt_out}" ]; then
echo "ERROR: Unable to find specified vertical coordinate file ${vrt_out}"
exit 1
fi # ! -f
vrt_opt="--vrt_out=${vrt_out}"
rmp_opt="--vrt_out=${vrt_out} ${rmp_opt}"
if [ -n "${vrt_xtr}" ]; then
if [ ${vrt_xtr} = 'mss_val' ] || [ ${vrt_xtr} = 'missing_value' ] || [ ${vrt_xtr} = 'msv' ]; then
vrt_opt="--rgr xtr_mth=mss_val ${vrt_opt}"
rmp_opt="--xtr_mth=mss_val ${rmp_opt}"
elif [ ${vrt_xtr} = 'nrs_ngh' ] || [ ${vrt_xtr} = 'nearest_neighbor' ] || [ ${vrt_xtr} = 'nn' ]; then
vrt_opt="--rgr xtr_mth=nrs_ngh ${vrt_opt}"
rmp_opt="--xtr_mth=nrs_ngh ${rmp_opt}"
else
echo "${spt_nm}: ERROR ${vrt_xtr} is not a valid extrapolation method"
echo "${spt_nm}: HINT Valid vertical extrapolation methods and synonyms are mss_val,missing_value,msv | nrs_ngh,nearest_neighbor,nn"
exit 1
fi # !vrt_xtr
fi # !vrt_xtr
rgr_opt="${rgr_opt} ${vrt_opt}"
fi # !vrt_out
yyyy_clm_srt=${yyyy_srt}
yyyy_clm_end=${yyyy_end}
yyyy_clm_srt_dec=${yyyy_srt}
yyyy_clm_end_dec=${yyyy_end}
mm_ann_srt='01' # [idx] First month used in annual climatology
mm_ann_end='12' # [idx] Last month used in annual climatology
mm_djf_srt='01' # [idx] First month used in DJF climatology
mm_djf_end='12' # [idx] Last month used in DJF climatology
yr_cln=${yr_nbr} # [nbr] Calendar years in climatology
if [ ${wnt_md} = 'djf' ]; then
yyyy_clm_srt_dec=`printf "%04d" ${yr_srtm1}`
yyyy_clm_end_dec=`printf "%04d" ${yr_endm1}`
mm_ann_srt='12'
mm_ann_end='11'
mm_djf_srt='12'
mm_djf_end='02'
let yr_cln=${yr_cln}+1
fi # !djf
# 20231221: Automatically generate input filename list based on command-line options
if [ ${inp_aut} = 'Yes' ] && [ ${clm_md} = 'mth' ] || [ ${clm_md} = 'ann' ]; then
fl_nbr=0
yr_srt_all=${yyyy_srt}
yr_end_all=${yyyy_end}
if [ ${wnt_md} = 'djf' ]; then
yr_srt_all=${yr_srtm1}
fi # !wnt_md
for yr in `seq ${yr_srt_all} ${yr_end_all}`; do
YYYY=`printf "%04d" ${yr}`
clm_idx=0
if [ "${clm_md}" = 'ann' ]; then
fl_in[${fl_nbr}]="${caseid}.${mdl_nm}.${hst_nm}.${YYYY}-${ann_sfx}.nc"
let fl_nbr=${fl_nbr}+1;
continue;
fi # !ann
for mth in {01..12}; do
let clm_idx=${clm_idx}+1
MM=`printf "%02d" ${clm_idx}`
# Only construct filenames for months that are used
if [ ${wnt_md} = 'djf' ] && [ ${yr} = ${yr_srt_all} ] && [ ${MM} != '12' ]; then continue; fi
if [ ${wnt_md} = 'djf' ] && [ ${yr} = ${yr_end_all} ] && [ ${MM} = '12' ]; then continue; fi
if [ ${mth_flg} = 'Yes' ] && [ ${yr} = ${yr_srt_all} ] && [ ${MM} -lt ${mth_srt_rth} ]; then continue; fi
if [ ${mth_flg} = 'Yes' ] && [ ${yr} = ${yr_end_all} ] && [ ${MM} -gt ${mth_end_rth} ]; then continue; fi
if [ ${nm_typ} = 'cesm' ]; then
fl_in[${fl_nbr}]="${caseid}.${mdl_nm}.${hst_nm}.${YYYY}-${MM}.nc"
elif [ ${nm_typ} = 'mpas' ]; then # Use MPAS not CESM conventions
if [ ${mdl_nm} = 'mali' ]; then
fl_in[${fl_nbr}]="${mdl_nm}.${hst_nm}.${YYYY}-${MM}-01_00000.nc"
else # !mdl_nm
# 20161130: Really old MPAS rule until today (for pre-v1 data)
# fl_in[${fl_nbr}]="${caseid}.${mdl_nm}.${YYYY}-${MM}-01_00.00.00.nc"
# 20240619: Newer though still old MPAS rule until today (for v1 data)
# Example file: /scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850S.ne30_oEC_ICG.edison/run/mpascice.hist.am.timeSeriesStatsMonthly.0001-02-01.nc
fl_in[${fl_nbr}]="${mdl_nm}.hist.am.timeSeriesStatsMonthly.${YYYY}-${MM}-01.nc"
# 20240619: Latest MPAS rule (for v2+ data) prepends caseid
if [ -n ${caseid} ]; then
fl_in[${fl_nbr}]="${caseid}.${fl_in[${fl_nbr}]}"
fi # !caseid
fi # !mdl_nm
elif [ ${nm_typ} = 'yyyymm' ]; then # Generate from caseid + YYYYMM
fl_in[${fl_nbr}]="${bs_nm}_${YYYY}${MM}.${bs_sfx}"
elif [ ${nm_typ} = 'yyyy-mm' ]; then # Generate from caseid + YYYY-MM
fl_in[${fl_nbr}]="${bs_nm}.${YYYY}-${MM}.${bs_sfx}"
elif [ ${nm_typ} = 'yyyy-mm-01' ]; then # Generate from caseid + YYYY-MM-01
fl_in[${fl_nbr}]="${bs_nm}.${YYYY}-${MM}-01.${bs_sfx}"
elif [ ${nm_typ} = 'yyyy-mm-01-00000' ]; then # Generate from caseid + YYYY-MM-01-00000
# fl_in[${fl_nbr}]="${caseid}.${mdl_nm}.${hst_nm}.${YYYY}-${MM}-01-00000.${bs_sfx}"
fl_in[${fl_nbr}]="${bs_nm}.${YYYY}-${MM}-01-00000.${bs_sfx}"
fi # !cesm
let fl_nbr=${fl_nbr}+1
done # !mth
done # !yr
fi # !inp_aut
if [ ${inp_glb} = 'Yes' ]; then
for fl in "${drc_in}"/*.nc "${drc_in}"/*.nc3 "${drc_in}"/*.nc4 "${drc_in}"/*.nc5 "${drc_in}"/*.nc6 "${drc_in}"/*.nc7 "${drc_in}"/*.cdf "${drc_in}"/*.hdf "${drc_in}"/*.he5 "${drc_in}"/*.h5 ; do
if [ -f "${fl}" ] || [ -L "${fl}" ]; then
fl_in[${fl_nbr}]=$(basename ${fl}) # NB: ncclimo uses basename($fl) here, and ppn_opt later, unlike ncremap
let fl_nbr=${fl_nbr}+1
fi # !file
done
fi # !inp_glb
if [ ${inp_psn} = 'Yes' ]; then
# Read any positional arguments
for ((psn_idx=1;psn_idx<=psn_nbr;psn_idx++)); do
fl_in[(${psn_idx}-1)]=${!psn_idx}
fl_nbr=${psn_nbr}
done # !psn_idx
fi # !inp_psn
if [ ${inp_std} = 'Yes' ]; then
# Input awaits on unit 0, i.e., on stdin
while read -r line; do # NeR05 p. 179
fl_in[${fl_nbr}]=${line}
let fl_nbr=${fl_nbr}+1
done < /dev/stdin
fi # !inp_std
# Sanity check that input files were found
if [ ${fl_nbr} -eq 0 ] || [ -z ${fl_in[0]} ]; then
printf "${spt_nm}: ERROR No input files were found: \$fl_nbr=${fl_nbr}, \${fl_in[0]}=\"${fl_in[0]}\". At least one input file is required. Please re-examine your input file specification.\n"
printf "The input method employed is the one whose value is \"Yes\": inp_aut = ${inp_aut}, inp_glb = ${inp_glb}, inp_psn = ${inp_psn}, inp_std = ${inp_std}.\n"
if [ ${inp_std} = 'Yes' ]; then
printf "${spt_nm}: HINT Expecting input from stdin. If you did not intend to provide input files on stdin, then perhaps ${spt_nm} a parent process (e.g., batch queue system or parent script) is controlling stdin. This can lead ${spt_nm} to expect input filenames on stdin. To prevent this from occurring, trying invoking with the --no_stdin switch, i.e., \'${spt_nm} --no_stdin ...\'.\n"
fi # inp_std
exit 1
fi # !fl_nbr
# Sanity check that monthly climatologies received a multiple of twelve files
# Automatically generated filenames have always adhered to this requirement
# 20250509: Provision of filenames via other methods never checked this until NCO 5.3.4
if [ ${clm_flg} = 'Yes' ] && [ ${clm_md} = 'mth' ]; then
fpy=12
let fl_rmd=${fl_nbr}%${fpy}
if [ ${fl_rmd} -ne 0 ]; then
printf "${spt_nm}: ERROR User requested climatology of monthly files and supplied ${fl_nbr} (not a multiple of 12) input files. Climatologies of monthly resolution data are only supported for whole years, not partial years (although the splitter works with partial years when the --mth_srt and --mth_end options are supplied).\n"
printf "${spt_nm}: HINT Provide input files in multiples of 12\n"
exit 1
fi # !fl_rmn
fi # !clm_flg, !clm_md
# Prepend drc_in to fl_in in MFOs (ncra, ncrcat)
# 20240401 Also prepend drc_in to fl_in in SFOs (ncks, ncap2) used in hfc/hfs modes and in rgn_stt
# Otherwise operators will fail to find input files when cwd != drc_in
ppn_opt="-p ${drc_in}"
# 20220111 If input files include absolute path, then use fl_in as-is later on
if [ "${fl_in[0]}" ]; then
if [ "$(basename ${fl_in[0]})" != "${fl_in[0]}" ] || [ "${fl_in[0]:0:1}" = '/' ]; then
ppn_opt=''
fi # !basename
fi # !fl_in[0]
if [ ${clm_md} = 'hfc' ] || [ ${clm_md} = 'hfs' ]; then
if [ -z "${tpd_usr}" ]; then
# User requested high-frequency mode and did not specify tpd so (try to) infer it
# Examine bounds of second timestep in case first timestep is instantaneous (e.g., restart values)
tm_var='time'
tm_bnd=`ncks --trd -M -m -v ${tm_var} ${ppn_opt} ${fl_in[0]} | grep -E -i "^${tm_var} attribute [0-9]+: bounds" | cut -f 11- -d ' ' | sed 's/^ *//g'`
if [ -z "${tm_bnd}" ]; then
# 20250509 Use time-coordinate values when time bounds DNE
tm_bnd_flg='No'
#printf "${spt_nm}: ERROR High-frequency mode needs but failed to find bounds variable for record coordinate named \"${tm_var}\" in input file \"${fl_in[0]}\". CF requires bounds variable name to be specified as value of \"bounds\" attribute of coordinate used for time-mean output. Typically record bounds variable is named \"time_bnds\" or \"time_bounds\".\nHINT: A workaround, that also works for temporally instantaneous data (e.g., cell_methods=\"time: point\") that lacks a time bounds variable, is to explicitly specify the number of timesteps-per-day in every input file with the --tpd=\$tpd option.\n"
#exit 1
fi # !tm_bnd
# 20201220: Using %g not %f in format results in tpd=10 not tpd=8 for 3-hourly EAM data!
# 20210815: Prefer to hyperslab second time index since some files contain an "instantaneous" (i.e., zero-duration) first timestep.
# However this presumes file has multiple timesteps---fails with monthly data
# 20210902: Use first time index if file has only one timestep
tm_nbr=`ncks --trd -M ${ppn_opt} ${fl_in[0]} | grep -E -i "^Root record dimension 0:" | cut -f 10- -d ' '`
if [ "${tm_bnd_flg}" = 'Yes' ]; then
[ "${tm_nbr}" -gt 1 ] 2>/dev/null && tm_idx='1' || tm_idx='0'
tpd_out=`ncks --trd -H -s '%25.15f ' -v ${tm_bnd} -d ${tm_var},${tm_idx} -C ${ppn_opt} ${fl_in[0]} | awk '{ print 1.0/($2-$1); exit }'`
else # !tm_bnd_flg
if [ "${tm_nbr}" -le 1 ]; then
printf "${spt_nm}: ERROR High-frequency mode needs to know temporal resolution in timesteps-per-day and user did not did not explicitly specify the number of timesteps-per-day with the --tpd=\$tpd option. Therefore ${spt_nm} attempted to infer it. No time bounds variable was found for the record coordinate named \"${tm_var}\" in input file \"${fl_in[0]}\", therefore ${spt_nm} attempted to infer tpd from the interval between the first two values of \"${tm_var}\". However the time coordinate has only one value in the first input file so no time interval could be computed.\nHINT: Put multiple timesteps in the first input file and/or add a time bounds variable to it.\n"
exit 1
fi # !tm_nbr
tpd_out=`ncks --trd -H -s '%25.15f ' -v ${tm_var} -d ${tm_var},0,1 -C ${ppn_opt} ${fl_in[0]} | awk '{ print 1.0/($2-$1); exit }'`
fi # !tm_bnd_flg
# Ensure result is positive integer
[ "${tpd_out}" -ge 0 ] 2>/dev/null && tpd_ntg='Yes' || tpd_ntg='No'
if [ "${tpd_ntg}" != 'Yes' ]; then
printf "${spt_nm}: ERROR Inferred value of tpd=${tpd_out} is not a positive integer. User requested high-frequency climatology or splitter operations without specifying the number of timesteps per day (tpd) in the input files, so ${spt_nm} tried to infer tpd from the first input file. This operation resulted in a nonsense value. This is not surprising since a number of assumptions must hold true for the inferral to work correctly.\nHINT: Explicitly specify the number of timesteps-per-day in every input file with the --tpd=\$tpd option.\n"
exit 1
fi # !tpd_ntg
fi # !tpd_usr
fi # !hfc && !hfs
# Parse grid/map arguments before in_fl arguments so we know whether this could be a map-only invocation
if [ "${tms_flg}" = 'Yes' ]; then
if [ -n "${var_lst}" ]; then
# 20210823: Pre-process extraction list if it contains a regular expression character
# Match var_lst against first file contents to create real extraction list
# Only necessary in subset mode to extract distinct timeseries
# https://stackoverflow.com/questions/26621736/how-to-check-if-a-string-contains-a-special-character
if [[ "${var_lst}" == *[\^\$\+\?\.\&\(\)\[\]\{\}\|]* ]]; then
var_lst=`ncks -C -v ${var_lst} --lst_xtr ${ppn_opt} ${fl_in[0]}`
fi # !rx
# Turn var_lst into array of excluded variables if so requested
if [ "${xcl_flg}" = 'Yes' ]; then
var_lst_xcl=( ${var_lst//,/ } )
fi # !xcl_flg
fi # !var_lst
# Create list of all variables when none specified, and array of all variables to construct exclusion list
if [ -z "${var_lst}" ] && [ "${xcl_flg}" = 'Yes' ]; then
echo "${spt_nm}: ERROR Exclude option (--xcl_var | --xcl | --exclude | --exclude_variables) only works in conjunction with explicitly specified variable extraction list"
echo "${spt_nm}: HINT Re-run with explicitly specified variable list (-v var_lst)"
exit 1
fi # !var_lst
if [ -z "${var_lst}" ] || [ "${xcl_flg}" = 'Yes' ]; then
if [ -z "${var_lst}" ]; then
echo "${spt_nm}: WARNING Splitter mode without explicitly specified variable list (i.e., -v var_lst) splits all variables of rank >= 2 into separate files, thus doubling the on-disk data amount"
fi # !var_lst
if [ "${rgn_stt}" != 'No' ]; then
# 20250327 Restrict regional statistics to horizontal-only (lat, lon, ncol) timeseries
# ncap2 regional timeseries will break when extra (e.g., vertical) dimensions are present
var_lst_flg_opt='--lst_hrz'
else # !rgn_stt
# Timeseries of any rank >= 2 are OK
var_lst_flg_opt='--lst_rnk_ge2'
fi # !rgn_stt
var_lst=`ncks ${var_lst_flg_opt} ${ppn_opt} ${fl_in[0]}`
if [ "${xcl_flg}" = 'Yes' ]; then
var_lst_all=( ${var_lst//,/ } )
fi # !xcl_flg
fi # !var_lst
# 20240617 Handle --xcl_flg for timeseries mode
if [ "${xcl_flg}" = 'Yes' ]; then
if bsh_mjr_vrs_lt_4; then
echo "${spt_nm}: ERROR Exclude flag (--xcl_var | --xcl | --exclude | --exclude_variables) in timeseries mode only works with Bash v. 4.0+ and this Bash is version ${BASH_VERSION}"
echo "${spt_nm}: HINT Re-run in Bash v. 4.0+ shell"
exit 1
fi # !bsh_mjr_vrs_lt_4()
if [ ${dbg_lvl} -ge 2 ]; then
printf "var_lst_all="
for var in ${var_lst_all[@]}; do
printf "${var},"
done # !var_lst_all
printf "\n"
fi # !dbg
if [ ${dbg_lvl} -ge 2 ]; then
printf "var_lst_xcl="
for var in ${var_lst_xcl[@]}; do
printf "${var},"
done # !var_lst_xcl
printf "\n"
fi # !dbg
# https://stackoverflow.com/questions/48791859/removing-items-from-one-list-when-present-in-another-in-bash
# Create associative array with variable names as keys, and a fixed value
declare -A var_lst_xtr=( ) # Requires Bash 4.0 or later
for var in "${var_lst_all[@]}"; do
var_lst_xtr[${var}]=1
done # !var_lst_all
# Remove keys associated with excluded variables
for var in "${var_lst_xcl[@]}"; do
unset "var_lst_xtr[${var}]"
done # !var_lst_xcl
if [ ${dbg_lvl} -ge 2 ]; then
printf "var_lst_xtr="
# Extraction list is the keys (not values) of this array
for var in ${!var_lst_xtr[@]}; do
printf "${var},"
done # !var_lst_xtr
printf "\n"
fi # !dbg
# Re-assemble extraction array into standard list
var_nbr=0 # [sng] Number of split (subset into timeseries) files
var_lst='' # [sng] Variables to process
for var in "${!var_lst_xtr[@]}"; do
var_lst="${var_lst},${var}"
let var_nbr=${var_nbr}+1
done # !var_lst_xtr
# Remove leading comma from list
var_lst="${var_lst:1}"
fi # !xcl_flg
# http://stackoverflow.com/questions/27702452/loop-through-a-comma-separated-shell-variable
var_nbr=0 # [sng] Number of split (subset into timeseries) files
for var in ${var_lst//,/ }; do
# NB:
var_sbs[${var_nbr}]=${var}
let var_nbr=${var_nbr}+1
done # !var_lst
if [ -z "${job_usr}" ]; then
job_nbr=${var_nbr}
fi # !job_usr
# Input files per year
flg_dct='Yes' # [flg] Double-check time in years against file count
if [ "${clm_md}" = 'ann' ]; then
fpy=1
elif [ "${clm_md}" = 'dly' ]; then
dpf=1
let fpy=${dpy}/${dpf}
elif [ "${clm_md}" = 'hfc' ]; then
fpy=12
flg_dct='No'
elif [ "${clm_md}" = 'hfs' ]; then
# CESM/E3SM high-frequency output often employs noleap calendar and 30 days per file (no matter how many timesteps per day)
if [ -z "${dpf}" ]; then
# 20210329 Automatically infer dpf from first input file
# E3SM/CESM high-frequency timeseries (e.g., h4 files) are all equal # timesteps, not duration
# First timestep in first file (often? always?) is of zero duration and contains the IC
# Remaining timesteps in first file, and all timesteps in following files, are of normal duration
# For example 3-hourly E3SM/CESM output files of 120 timesteps length are each 30 days duration except the first file which is only 29.875 days because its first timestep is of zero duration
# Solution: Base dpf on second file, not first file, if it exists
fl_idx=$((fl_nbr > 1 ? 1 : 0))
tm_var='time'
tm_bnd=`ncks --trd -M -m -v ${tm_var} ${ppn_opt} ${fl_in[${fl_idx}]} | grep -E -i "^${tm_var} attribute [0-9]+: bounds" | cut -f 11- -d ' ' | sed 's/^ *//g'`
if [ -z "${tm_bnd}" ]; then
# 20250509 Use time-coordinate values when time bounds DNE
tm_bnd_flg='No'
fi # !tm_bnd
if [ "${tm_bnd_flg}" = 'Yes' ]; then
dpf=`ncks --trd -H -s '%25.15f ' -v ${tm_bnd} -d ${tm_var},0 -d ${tm_var},-1 -C ${ppn_opt} ${fl_in[${fl_idx}]} | awk '{ print $4-$1; exit }'`
else # !tm_bnd_flg
dpf=`ncks --trd -H -s '%25.15f ' -v ${tm_var} -d ${tm_var},0 -d ${tm_var},-1 -C ${ppn_opt} ${fl_in[${fl_idx}]} | awk '{ printf "%.0f\n",$2-$1; exit }'`
# This is a time coordinate not a time bounds so last-first value is often a decimal fraction not an integer
# For example, output files at 3 hourly resolution often have multiples of 8 timesteps with first at
# dayM day-fraction 0.0, last at dayN day-fraction 0.875, and difference is integer N-M plus 0.875.
# When difference > 0.5 it is often correct to round-up, i.e., to integer dpf=1.
fi # !tm_bnd_flg
# Ensure result is positive integer
[ "${dpf}" -ge 0 ] 2>/dev/null && dpf_ntg='Yes' || dpf_ntg='No'
if [ "${dpf_ntg}" != 'Yes' ]; then
printf "${spt_nm}: ERROR Inferred value of dpf=${dpf} is not a positive integer. User requested high-frequency splitter operations without specifying the number of days-per-file (dpf) in the input files, so ${spt_nm} tried to infer dpf from the first input file. This operation resulted in a nonsense value. This is not surprising since a number of assumptions must hold true for the inferral to work correctly.\nHINT: Explicitly specify the number of days-per-file in every input file with the --dpf option.\n"
exit 1
fi # !dpf_ntg
fi # !dpf
# 20220818
let fpy=${dpy}/${dpf}
#fpy=$(echo "${dpy}/${dpf}" | bc -l) # NB: dpy/dpf more accurate than fpy
flg_dct='No'
# !clm_md=hfs
elif [ "${clm_md}" = 'mth' ]; then
fpy=12
fi # !clm_md
if [ "${clm_md}" = 'hfs' ]; then
let yr_sbs=${yr_nbr}
else
let yr_sbs=${fl_nbr}/${fpy}
let fl_rmd=${fl_nbr}%${fpy}
fi # !clm_md
if [ "${flg_dct}" = 'Yes' ]; then
if [ "${mth_flg}" = 'Yes' ]; then
if [ ${mth_nbr} -ne ${fl_nbr} ]; then
printf "${spt_nm}: ERROR Number of input files = ${fl_nbr} differs from number of months = ${mth_nbr} computed from calendar inputs\n"
printf "${spt_nm}: HINT In this mode (clm_md=${clm_md}) splitter expects one monthly file per calendar month\n"
exit 1
fi # !mth_nbr
else # !mth_flg
if [ ${fl_rmd} -ne 0 ]; then
printf "${spt_nm}: ERROR ${fl_nbr} files of clm_md=${clm_md} input contain non-integral number of years, ${fl_rmd} files leftover in final year\n"
printf "${spt_nm}: HINT Provide input filenames in multiples of ${fpy}\n"
exit 1
fi # !fl_rmd
if [ ${yr_sbs} -ne ${yr_nbr} ]; then
# Sanity check that number of files specified matches number expected from date switches
printf "${spt_nm}: ERROR The ${fl_nbr} files specified (via stdin pipe, positional, or input directory) are expected to contain ${yr_sbs} years of data whereas the date options specify ${yr_nbr} years of data\n"
printf "${spt_nm}: HINT Number of files at ${fpy} files-per-year must match number of years implied by arguments to start- and end-year switches (--yr_srt=${yr_srt} and --yr_end=${yr_end}, respectively). Check that the stdin pipe, positional arguments, or globbed input directory contain the intended number of filenames.\n"
exit 1
fi # !yr_sbs
fi # !mth_flg
fi # !flg_dct
# How many segments of output?
if [ "${mth_flg}" = 'Yes' ]; then
# NB: When start/end months are not Jan/Dec, sgm_rmd measured in months
let mpf_max=${fpy}*${ypf_max}
let sgm_nbr=${mth_nbr}/${mpf_max}
let sgm_rmd=${mth_nbr}%${mpf_max}
if [ ${sgm_rmd} -ne 0 ]; then
let sgm_nbr=${sgm_nbr}+1
rmd_flg='Yes'
else # !sgm_rmd
rmd_flg='No'
fi # !sgm_rmd
else # !mth_flg
# NB: When start/end months are Jan/Dec, sgm_rmd measured in years
let sgm_nbr=${yr_sbs}/${ypf_max}
let sgm_rmd=${yr_sbs}%${ypf_max}
if [ ${sgm_rmd} -ne 0 ]; then
let sgm_nbr=${sgm_nbr}+1
rmd_flg='Yes'
else # !sgm_rmd
rmd_flg='No'
fi # !sgm_rmd
fi # !mth_flg
let sgm_nbrm1=${sgm_nbr}-1
if [ "${rgn_stt}" != 'No' ]; then
# Export timeseries of regional/global statistics
# MUST pass rgn_stt to cmd_rgn script as an enum/flag (i.e., flg_sum) not a string. Sigh...
if [ "${rgn_stt}" = 'sum' ] || [ "${rgn_stt}" = 'ttl' ] || [ "${rgn_stt}" = 'total' ] || [ "${rgn_stt}" = 'integral' ]; then
rgn_stt='sum'
flg_sum=1
elif [ "${rgn_stt}" = 'avg' ] || [ "${rgn_stt}" = 'average' ] || [ "${rgn_stt}" = 'mean' ]; then
rgn_stt='avg'
flg_sum=0
else
echo "${spt_nm}: ERROR \"${rgn_stt}\" is an invalid value for regional_statistic"
echo "${spt_nm}: HINT The valid values for regional_statistic are \"avg\" (or synonyms \"average\", and \"mean\") and \"sum\" (or synonyms \"ttl\", \"total\", and \"integral\")"
exit 1
fi # !rgn_stt
if [ "${prc_mpas}" = 'Yes' ]; then
# 20240619
# MPAS-LI, MPAS-O, MPAS-SI v2+ datasets omit latitude variable
lat_nm='latCell' # Generic MPAS latitude variable
# MPAS-A, MPAS-LI, MPAS-O, MPAS-SI v2+ datasets omit (time-constant) area variable
# NB: areaCell is full gridcell area on MPAS mesh, and does not account for landfrac/ocnfrac
# Hence, area_wgt variable diagnosed below will be inaccurate for MPAS-O unless 'ocnfrac' is given and applied as sgs_frc
area_nm='areaCell' # Generic MPAS fixed grid area variable
if [ "${prc_mpasseaice}" = 'Yes' ]; then
# MPAS-SI v2+ datasets omit latitude though include (time-varying) area variable
sgs_frc='timeMonthly_avg_iceAreaCell'
fi # !prc_mpasseaice
echo "WARNING: Regional average timeseries option requires latitude and area variables. These variables are not present in many MPAS datasets."
echo "HINT: First append latitude and area variables to raw timeseries datasets with, e.g., ncks -C -v ${lat_nm},${area_nm} in.nc out.nc"
fi # !prc_mpas
# 20191217 Regional average timeseries require area and latitude variables
var_xtr="${var_xtr},${area_nm},${lat_nm}"
if [ "${prc_typ}" = 'clm' ] || [ "${prc_typ}" = 'elm' ]; then
sgs_frc='landfrac'
fi # !prc_typ
if [ -n "${sgs_frc}" ]; then
area_wgt="${area_nm}*${sgs_frc}"
var_xtr="${var_xtr},${sgs_frc}"
else
area_wgt="${area_nm}"
fi # !sgs_frc
# 20240329 Simplify life by automatically determining hrz_dmn from area_nm in first file
# Works for EAM, ELM, unstructured, structured, ...
# https://unix.stackexchange.com/questions/108250/print-the-string-between-two-parentheses
#hrz_dmn=`ncks -C -m -v ${area_nm} ${ppn_opt} ${fl_in[0]} | grep "${area_nm}\(" | sed 's/^.*(//;s/) ;$//'` # Works at prompt, not in script
hrz_dmn=`ncks -C -m -v ${area_nm} ${ppn_opt} ${fl_in[0]} | grep -E "float.*${area_nm}|double.*${area_nm}" | sed 's/^.*(//;s/) ;$//'`
if [ -n "${hrz_dmn}" ]; then
# 20240329 Prepend string literal '$' to each horizontal dimension for use in argument to ncap2 avg() function (e.g., 'lat,lon' -> '$lat,$lon')
hrz_arg="\$${hrz_dmn/,/,\$}"
fi # !hrz_dmn
fi # !rgn_stt
else # !tms_flg
if [ "${mth_flg}" = 'Yes' ]; then
echo "${spt_nm}: ERROR Monthly climo-mode only supports climatologies that start in January (jfd-mode, the default) or December (djf) mode. Climo mode does not support arbitrary starting months (--mth_srt and --mth_end). If you would like ${spt_nm} to generate climos for arbitrary starting months, please talk to Charlie."
exit 1
fi # !mth_flg
if [ "${rgn_stt}" != 'No' ]; then
echo "${spt_nm}: ERROR Regional/global statistical timeseries functionality is supported for time-series (splitting) mode only. If you would like ${spt_nm} to provide global and regional statistical output for other types of climatologies, please talk to Charlie."
echo "${spt_nm}: HINT Create global statistics of climos with, e.g., \"ncwa -w ${area_nm} in.nc out.nc\""
echo "${spt_nm}: HINT See averaging documentation at http://nco.sf.net/nco.html#ncwa"
exit 1
fi # !rgn_stt
if [ -z "${out_nm}" ]; then
echo "${spt_nm}: ERROR Missing information needed to generate output filenames"
echo "${spt_nm}: HINT Generating input filenames for climos requires that users specify a case ID with -c \$caseid or specify with -m \$mdl_nm a recognized MPAS model name (like \"mpaso\"). Output filenames for climos are based on these inputs as well, or on the family name if specified with --fml_nm=\$fml_nm. When input filenames are not internally generated, but instead provided via stdin, positional arguments, or globbing, then the output filenames will be based on the family name or caseid, or model name, in that order of precedence."
echo "${spt_nm}: HINT ${spt_nm} needs this information to generate output filenames"
echo "${spt_nm}: HINT See invocation examples at http://nco.sf.net/nco.html#ncclimo"
exit 1
fi # out_nm
# 20211118 fxm: re-do these messages
if [ ${inp_std} = 'Yes' ] && [ ${clm_md} = 'ann' ]; then
echo "${spt_nm}: ERROR Detected input on pipe to stdin in annual climatology generation mode"
echo "${spt_nm}: HINT Piping filenames to ${spt_nm} only works when splitting files or in daily, monthly, or high-frequency climatology mode"
echo "${spt_nm}: HINT In annual climo generation mode, one must specify the year/month boundaries and ${caseid}, and from these ${spt_nm} will automatically generate the correct input file names"
echo "${spt_nm}: HINT See invocation examples at http://nco.sf.net/nco.html#ncclimo"
exit 1
fi # !stdin
fi # !tms_flg
# 20170807 Custom seasons
if [ "${csn_flg}" = 'Yes' ]; then
if [ "${clm_md}" != 'hfc' ] && [ "${clm_md}" != 'mth' ]; then
echo "${spt_nm}: ERROR Custom seasons available only in high-frequency or monthly climatology mode"
echo "${spt_nm}: HINT Re-run with --clm_md=hfc or --clm_md=mth or without --csn_lst/--seasons"
exit 1
fi # !mth
# Standard and custom season definitions
# Index into season definition table
csn_idx_srt=0 # [idx] Starting index for season definitions
csn_mam=0
csn_jja=1
csn_son=2
csn_djf=3
csn_jfm=4
csn_amj=5
csn_jas=6
csn_ond=7
csn_on=8
csn_fm=9
csn_ann=10
csn_nbr_max=11 # [nbr] Maximum number of seasons in definitions database
# Seasonal abbreviations, uppercase
csn_abb[${csn_mam}]='MAM'
csn_abb[${csn_jja}]='JJA'
csn_abb[${csn_son}]='SON'
csn_abb[${csn_djf}]='DJF'
csn_abb[${csn_jfm}]='JFM'
csn_abb[${csn_amj}]='AMJ'
csn_abb[${csn_jas}]='JAS'
csn_abb[${csn_ond}]='OND'
csn_abb[${csn_on}]='ON'
csn_abb[${csn_fm}]='FM'
csn_abb[${csn_ann}]='ANN'
# Seasonal abbreviations, lowercase
csn_abb_lc[${csn_mam}]='mam'
csn_abb_lc[${csn_jja}]='jja'
csn_abb_lc[${csn_son}]='son'
csn_abb_lc[${csn_djf}]='djf'
csn_abb_lc[${csn_jfm}]='jfm'
csn_abb_lc[${csn_amj}]='amj'
csn_abb_lc[${csn_jas}]='jas'
csn_abb_lc[${csn_ond}]='ond'
csn_abb_lc[${csn_on}]='on'
csn_abb_lc[${csn_fm}]='fm'
csn_abb_lc[${csn_ann}]='ann'
# Seasonal start-month index (1-based calendar)
csn_srt_idx[${csn_mam}]=3
csn_srt_idx[${csn_jja}]=6
csn_srt_idx[${csn_son}]=9
csn_srt_idx[${csn_djf}]=12
csn_srt_idx[${csn_jfm}]=1
csn_srt_idx[${csn_amj}]=4
csn_srt_idx[${csn_jas}]=7
csn_srt_idx[${csn_ond}]=10
csn_srt_idx[${csn_on}]=10
csn_srt_idx[${csn_fm}]=2
csn_srt_idx[${csn_ann}]=1
# Seasonal end-month index (1-based calendar)
csn_end_idx[${csn_mam}]=5
csn_end_idx[${csn_jja}]=8
csn_end_idx[${csn_son}]=11
csn_end_idx[${csn_djf}]=2
csn_end_idx[${csn_jfm}]=3
csn_end_idx[${csn_amj}]=6
csn_end_idx[${csn_jas}]=9
csn_end_idx[${csn_ond}]=12
csn_end_idx[${csn_on}]=11
csn_end_idx[${csn_fm}]=3
csn_end_idx[${csn_ann}]=12
for ((csn_idx=0;csn_idx<${csn_nbr_max};csn_idx++)); do
csn_srt_mm[${csn_idx}]=`printf "%02d" ${csn_srt_idx[${csn_idx}]}`
csn_end_mm[${csn_idx}]=`printf "%02d" ${csn_end_idx[${csn_idx}]}`
done # !csn_idx
# Seasonal number of months
csn_nom[${csn_mam}]=3
csn_nom[${csn_jja}]=3
csn_nom[${csn_son}]=3
csn_nom[${csn_djf}]=3
csn_nom[${csn_jfm}]=3
csn_nom[${csn_amj}]=3
csn_nom[${csn_jas}]=3
csn_nom[${csn_ond}]=3
csn_nom[${csn_on}]=2
csn_nom[${csn_fm}]=2
csn_nom[${csn_ann}]=12
# Seasonal monthly weights (ncra/nces -w argument), assumes noleap calendar = 365 dpy
csn_wgt[${csn_mam}]='31,30,31'
csn_wgt[${csn_jja}]='30,31,31'
csn_wgt[${csn_son}]='30,31,30'
csn_wgt[${csn_djf}]='31,31,28'
csn_wgt[${csn_jfm}]='31,28,31'
csn_wgt[${csn_amj}]='30,31,30'
csn_wgt[${csn_jas}]='31,31,30'
csn_wgt[${csn_ond}]='31,30,31'
csn_wgt[${csn_on}]='31,30'
csn_wgt[${csn_fm}]='28,31'
csn_wgt[${csn_ann}]='31,28,31,30,31,30,31,31,30,31,30,31'
# Which seasons are requested?
# http://stackoverflow.com/questions/27702452/loop-through-a-comma-separated-shell-variable
csn_nbr=0 # [sng] Number of seasons to compute
for csn in ${csn_lst//,/ }; do
csn_rqs[${csn_nbr}]=${csn}
# NB: Requested seasons are 0-based, defined seasons are 0-based
for ((csn_dfn_idx=${csn_idx_srt};csn_dfn_idx<${csn_nbr_max};csn_dfn_idx++)); do
if [[ "${csn}" =~ "${csn_abb[${csn_dfn_idx}]}" ]] || [[ "${csn}" =~ "${csn_abb_lc[${csn_dfn_idx}]}" ]]; then
# Map requested to defined (r2d) seasons and inverse (d2r)
# map_r2d[0]=3 means first season that user requested (i.e., in csn_lst) is fourth defined in table
# map_d2r[3]=0 means fourth defined season is first requested
map_r2d[${csn_nbr}]=${csn_dfn_idx}
map_d2r[${csn_dfn_idx}]=${csn_nbr}
let csn_nbr=${csn_nbr}+1
break
fi # !match
done # !csn_dfn_idx
if [ "${csn_dfn_idx}" -eq "${csn_nbr_max}" ]; then
printf "${spt_nm}: ERROR Requested season ${csn} not defined\n"
exit 1
fi # !match
done # !csn_lst
# Were all four seasons requested so annual mean can be computed from seasons?
ann_flg='No'
ann_nbr=0
if [[ ${csn_lst} =~ 'mam' ]] || [[ ${csn_lst} =~ 'MAM' ]]; then
let clm_csn_mam_idx=${map_d2r[${csn_mam}]}+12+1
if [[ ${csn_lst} =~ 'jja' ]] || [[ ${csn_lst} =~ 'JJA' ]]; then
let clm_csn_jja_idx=${map_d2r[${csn_jja}]}+12+1
if [[ ${csn_lst} =~ 'son' ]] || [[ ${csn_lst} =~ 'SON' ]]; then
let clm_csn_son_idx=${map_d2r[${csn_son}]}+12+1
if [[ ${csn_lst} =~ 'djf' ]] || [[ ${csn_lst} =~ 'DJF' ]]; then
let clm_csn_djf_idx=${map_d2r[${csn_djf}]}+12+1
ann_flg='Yes'
ann_nbr=1
fi # !djf
fi # !son
fi # !jja
fi # !mam
# Account for season number in climatology command number and parallelism
let clm_csn_srt_idx=12+1
let clm_csn_end_idx=12+${csn_nbr}
let clm_nbr=12+${csn_nbr}+${ann_nbr}
fi # !csn_flg
if [ "${mpi_flg}" = 'Yes' ]; then
if [ -n "${PBS_NODEFILE}" ]; then
nd_fl="${PBS_NODEFILE}"
elif [ -n "${SLURM_NODELIST}" ]; then
# SLURM returns compressed lists (e.g., "nid00[076-078,559-567]")
# Convert this to file with uncompressed list (like PBS)
# http://www.ceci-hpc.be/slurm_faq.html#Q12
# Save file in writable directory
nd_fl="${drc_out}/${spt_nm}.slurm_nodelist.pid${spt_pid}.tmp"
nd_lst=`scontrol show hostname ${SLURM_NODELIST}`
echo ${nd_lst} > ${nd_fl}
else
echo "${spt_nm}: ERROR MPI master process unable to find node-list for distributing jobs"
echo "${spt_nm}: ${spt_nm} uses first node-list found in \$PBS_NODEFILE or \$SLURM_NODELIST"
echo "${spt_nm}: However, none of these environment variables are set so there is no node-list for distributing MPI jobs"
echo "${spt_nm}: HINT: Requesting MPI-parallelism (i.e., invoking ${spt_nm} with \"-p mpi\") in a non-MPI environment will trigger this error. Use \"-p mpi\" only when one of the preceding schedulers has allocated (for interactive use) or will allocate (for non-interactive use) the compute nodes. Otherwise use the default background parallelism (use \"-p bck\" or omit the option completely) or use serial mode (use \"-p serial\"). See http://nco.sf.net/nco.html#par_typ for more information on parallelism."
exit 1
fi # !PBS
if [ "${tms_flg}" = 'Yes' ]; then
mpi_nbr=${var_nbr}
else
mpi_nbr=${clm_nbr}
fi # !tms_flg
# 20210310 MPI run command may need/use thr_nbr so set it first
if [ -z "${thr_usr}" ]; then
if [ -n "${PBS_NUM_PPN}" ]; then
# NB: use export OMP_NUM_THREADS when thr_nbr > 8
# thr_nbr=${PBS_NUM_PPN}
thr_nbr=$((PBS_NUM_PPN > 8 ? 8 : PBS_NUM_PPN))
fi # !pbs
fi # !thr_usr
if [ -n "${nd_fl}" ]; then
# NB: nodes are always 0-based, e.g., [0..11]
# For climo generation MPI index loops over months and is 1-based, e.g., [1..17] (December is 12 and ANN is 17)
# For climo subsetting MPI index loops over variables and is 0-based, e.g., [0..5], as are input files
nd_idx=0
for nd in `cat ${nd_fl} | uniq` ; do
nd_nm[${nd_idx}]=${nd}
let nd_idx=${nd_idx}+1
done # !nd
nd_nbr=${#nd_nm[@]}
if [ "${nd_nbr}" -eq 0 ]; then
echo "${spt_nm}: ERROR MPI-mode node number nd_nbr = ${nd_nbr}"
echo "${spt_nm}: HINT Parsing the node-list for distributing MPI jobs failed"
exit 1
fi # !nd_nbr
# NB: ncclimo and ncremap employ different node-allocation algorithms:
# ncclimo (monthly climatology mode) assigns monthly regridding and seasonal climos to different nodes (i.e., load-balances), and likewise for seasonal-regridding and annual climo
# ncclimo (splitter mode) uses simple round robin based on position in variable list
# ncremap uses simple round-robin allocation based on position in input file list
# Only ncclimo monthly climatology-mode uses 1-based cmd_mpi array
# ncclimo splitter-mode, and daily and annual climatology-mode, and ncremap all use 0-based cmd_mpi array
# Copy host-specific mpirun syntax but not node-allocation algorithms or loop indices between ncclimo and ncremap
# 20160502: Remove tasks-per-node limits (ntasks, npernode) so round-robin algorithm can schedule multiple jobs on same node
for ((mpi_idx_zro=0;mpi_idx_zro<mpi_nbr;mpi_idx_zro++)); do
mpi_idx=${mpi_idx_zro}
if [ "${clm_flg}" = 'Yes' ] && [ "${clm_hfc_or_mth}" = 'Yes' ]; then
# Offset MPI index from 0- to 1-based for traditional monthly-based climo generation
let mpi_idx=${mpi_idx_zro}+1
fi # !tms_flg
case "${HOSTNAME}" in
andes* | blues* | blogin* | b[0123456789][0123456789][0123456789] | chrysalis* | chrlogin* | chr-[0123456789][0123456789][0123456789][0123456789] | compy* | constance* | frontier* | login[0123456789][0123456789] | nid* | node* | perlmutter* )
# 20220518: Non-interactive batch jobs at NERSC/Perlmutter return HOSTNAME as login??, and nodes as nid??????
# 20160803: Non-interactive batch jobs at PNNL constance return HOSTNAME as node*, not constance*
# 20190526: Non-interactive batch jobs at PNNL compy return HOSTNAME as n????, not compy*
# 20210310: srun option<->long option equivalences are -N = --nodes, -n = --ntasks, -c = --cpus-per-task
# NB: NERSC staff say srun automatically assigns to unique nodes even without "-L $node" argument?
cmd_mpi[${mpi_idx}]="srun --nodelist ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} --nodes=1" ; ;; # SLURM
# cmd_mpi[${mpi_idx}]="srun --nodelist ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} --nodes=1 --ntasks=1" ; ;; # SLURM
# cmd_mpi[${mpi_idx}]="srun --nodelist ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} --nodes=1 --ntasks=1 --cpus-per-task=1" ; ;; # SLURM
derecho* )
# 20180120: Non-interactive batch jobs at NCAR return HOSTNAME as derecho?
# Derecho prefers 'mpiexec_mpt dplace -s 1 ncclimo ...'
# Unsure how to specify nd_nm to mpiexec_mpt
# mpirun from SGI MPT does not accept '-H nd_nm', unlike regular PBS
# PBSPro considers ncclimo a 'non-MPT application' so must set MPI_SHEPHERD=true
export MPI_SHEPHERD=true
# cmd_mpi[${mpi_idx}]="mpiexec_mpt dplace ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} -n 1" ; ;; # PBSPro
cmd_mpi[${mpi_idx}]="mpirun ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} -n 1" ; ;; # PBSPro
* )
# 20250122: Shift default node manager from mpirun to srun
cmd_mpi[${mpi_idx}]="srun --nodelist ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} --nodes=1" ; ;; # SLURM
# cmd_mpi[${mpi_idx}]="mpirun -H ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} -n 1" ; ;; # Other (PBS)
# cmd_mpi[${mpi_idx}]="mpirun -H ${nd_nm[$((${mpi_idx_zro} % ${nd_nbr}))]} -npernode 1 -n 1" ; ;; # Other
esac # !HOSTNAME
case "${HOSTNAME}" in
perlmutter* | login[0123456789][0123456789] | nid* )
# 20210319: Multiple srun commands cannot run concurrently on single Cori node without special options
# --gres=craynetwork:0 --mem=20000 tested by me, Noel Keen documented here:
# https://docs.nersc.gov/jobs/examples/#multiple-parallel-jobs-while-sharing-nodes
# These options are only expected to work on NERSC Cori
# craynetwork:0 (instead of, e.g., :1) allows any number of MPI jobs to run
# 20240406 Deprecate --gres=craynetwork:0 since not available on Perlmutter, and keep --mem option
# Argument to --mem is requested number of MB RAM per job so 96 GB Cori node with 4 MPI processes can use ~20000 MB RAM per job, and, I think, jobs in excess of that will queue until more RAM becomes available as old jobs finish
# 20210319: --zonesort=off tested by me does eliminate zonesort messages like these:
# slurmstepd: error: Detected zonesort setup failure: zonesort interface write failure (40558186.7)
# However, using zonesort option appears to slow-down throughput considerably
cmd_mpi[${mpi_idx}]="${cmd_mpi[${mpi_idx}]} --mem=${mem_mb}" ; ;; # SLURM
# cmd_mpi[${mpi_idx}]="${cmd_mpi[${mpi_idx}]} --nodes=1 --gres=craynetwork:0 --mem=${mem_mb} --zonesort=off" ; ;; # SLURM
esac # !HOSTNAME
done # !mpi_idx_zro
if [ -n "${SLURM_NODELIST}" ]; then
/bin/rm -f ${nd_fl}
fi # !SLURM
else # !nd_fl
mpi_flg='No'
for ((mpi_idx=0;mpi_idx<=mpi_nbr;mpi_idx++)); do
cmd_mpi[${mpi_idx}]=''
done # !mpi_idx
fi # !nd_fl
if [ -z "${job_usr}" ]; then
job_nbr=${nd_nbr}
fi # !job_usr
fi # !mpi_flg
if [ "${clm_flg}" = 'Yes' ] && [ "${clm_hfc_or_mth}" = 'Yes' ] && [ "${bnr_flg}" = 'No' ]; then
# For regular monthly climos, override default job_nbr (2) with months-per-year
# If user does not explicitly set job number then, for monthly climos, base it on parallelism type
if [ -z "${job_usr}" ]; then
if [ "${par_typ}" = ${par_bck} ]; then
# Background mode will run one batch with all twelve months
job_nbr=12
elif [ "${par_typ}" = ${par_mpi} ]; then
# MPI mode is conservative and will only use one job-per-node unless told otherwise
job_nbr=${nd_nbr}
elif [ "${par_typ}" = 'nil' ] || [ -z "${par_typ}" ]; then
# Serial mode equates to job_nbr=1, full serial
job_nbr=1
fi # !job_usr
fi # !job_usr
fi # !clm_md !bnr_flg
# Print initial state
if [ ${dbg_lvl} -ge 2 ]; then
printf "dbg: area_nm = ${area_nm}\n"
printf "dbg: area_wgt = ${area_wgt}\n"
printf "dbg: bnr_flg = ${bnr_flg}\n"
printf "dbg: bs_nm = ${bs_nm}\n"
printf "dbg: bs_sfx = ${bs_sfx}\n"
printf "dbg: caseid = ${caseid}\n"
printf "dbg: cf_flg = ${cf_flg}\n"
printf "dbg: clm_flg = ${clm_flg}\n"
printf "dbg: clm_md = ${clm_md}\n"
printf "dbg: clm_nbr = ${clm_nbr}\n"
printf "dbg: cmp_sng = ${cmp_sng}\n"
printf "dbg: csn_lst = ${csn_lst}\n"
printf "dbg: csn_nbr = ${csn_nbr}\n"
printf "dbg: d2f_flg = ${d2f_flg}\n"
printf "dbg: d2f_opt = ${d2f_opt}\n"
printf "dbg: dbg_lvl = ${dbg_lvl}\n"
printf "dbg: dfl_lvl = ${dfl_lvl}\n"
printf "dbg: dpt_opt = ${dpt_opt}\n"
printf "dbg: drc_in = ${drc_in}\n"
printf "dbg: drc_nco = ${drc_nco}\n"
printf "dbg: drc_out = ${drc_out}\n"
printf "dbg: drc_prv = ${drc_prv}\n"
printf "dbg: drc_pwd = ${drc_pwd}\n"
printf "dbg: drc_rgr = ${drc_rgr}\n"
printf "dbg: drc_spt = ${drc_spt}\n"
printf "dbg: drc_xtn = ${drc_xtn}\n"
printf "dbg: fl_fmt = ${fl_fmt}\n"
printf "dbg: fl_nbr = ${fl_nbr}\n"
printf "dbg: flg_sum = ${flg_sum}\n"
printf "dbg: fml_nm = ${fml_nm}\n"
printf "dbg: gaa_sng = ${gaa_sng}\n"
printf "dbg: hdr_pad = ${hdr_pad}\n"
printf "dbg: hrd_pth = ${hrd_pth}\n"
printf "dbg: hrz_arg = ${hrz_arg}\n"
printf "dbg: hrz_dmn = ${hrz_dmn}\n"
printf "dbg: hst_nm = ${hst_nm}\n"
printf "dbg: inp_aut = ${inp_aut}\n"
printf "dbg: inp_glb = ${inp_glb}\n"
printf "dbg: inp_psn = ${inp_psn}\n"
printf "dbg: inp_std = ${inp_std}\n"
printf "dbg: job_nbr = ${job_nbr}\n"
printf "dbg: lnk_flg = ${lnk_flg}\n"
printf "dbg: mdl_nm = ${mdl_nm}\n"
printf "dbg: mdl_typ = ${mdl_typ}\n"
printf "dbg: mpi_flg = ${mpi_flg}\n"
printf "dbg: mpi_nbr = ${mpi_nbr}\n"
printf "dbg: mth_end = ${mth_end}\n"
printf "dbg: mth_nbr = ${mth_nbr}\n"
printf "dbg: mth_srt = ${mth_srt}\n"
printf "dbg: nco_opt = ${nco_opt}\n"
printf "dbg: ncr_flg = ${ncr_flg}\n"
printf "dbg: nd_nbr = ${nd_nbr}\n"
printf "dbg: nm_typ = ${nm_typ}\n"
printf "dbg: no_ntv = ${no_ntv_tms}\n"
printf "dbg: out_nm = ${out_nm}\n"
printf "dbg: par_typ = ${par_typ}\n"
printf "dbg: ppn_opt = ${ppn_opt}\n"
printf "dbg: prc_opt = ${prc_opt}\n"
printf "dbg: prc_typ = ${prc_typ}\n"
printf "dbg: qnt_prc = ${qnt_prc}\n"
printf "dbg: rgn_avg = ${rgn_avg}\n"
printf "dbg: rgn_stt = ${rgn_stt}\n"
printf "dbg: rgr_map = ${rgr_map}\n"
printf "dbg: rgr_opt = ${rgr_opt}\n"
printf "dbg: rgr_sfx = ${rgr_sfx}\n"
printf "dbg: sgm_nbr = ${sgm_nbr}\n"
printf "dbg: sgm_rmd = ${sgm_rmd}\n"
printf "dbg: sgs_frc = ${sgs_frc}\n"
printf "dbg: sum_scl = ${sum_scl}\n"
printf "dbg: thr_nbr = ${thr_nbr}\n"
printf "dbg: tm_bnd = ${tm_bnd}\n"
printf "dbg: tm_nbr = ${tm_nbr}\n"
printf "dbg: tm_var = ${tm_var}\n"
printf "dbg: tms_flg = ${tms_flg}\n"
printf "dbg: tpd_out = ${tpd_out}\n"
printf "dbg: uio_flg = ${uio_flg}\n"
printf "dbg: var_lst = ${var_lst}\n"
printf "dbg: var_xtr = ${var_xtr}\n"
printf "dbg: vrt_out = ${vrt_out}\n"
printf "dbg: vrt_xtr = ${vrt_xtr}\n"
printf "dbg: wnt_md = ${wnt_md}\n"
printf "dbg: xcl_flg = ${xcl_flg}\n"
printf "dbg: xtn_flg = ${xtn_flg}\n"
printf "dbg: ypf_max = ${ypf_max}\n"
printf "dbg: yr_sbs = ${yr_sbs}\n"
printf "dbg: yyyy_end = ${yyyy_end}\n"
printf "dbg: yyyy_srt = ${yyyy_srt}\n"
if [ "${csn_flg}" = 'Yes' ]; then
for ((csn_idx=0;csn_idx<${csn_nbr};csn_idx++)); do
printf "dbg: Requested season index ${csn_idx} (\"${csn_rqs[${csn_idx}]}\") is defined season index ${map_r2d[${csn_idx}]} (\"${csn_abb[${map_r2d[${csn_idx}]}]}\")\n"
done # !csn_idx
fi # !csn
fi # !dbg
if [ ${dbg_lvl} -ge 2 ]; then
printf "dbg: yyyy_srt = ${yyyy_srt}\n"
printf "dbg: yr_srt_rth = ${yr_srt_rth}\n"
printf "dbg: yr_srtm1 = ${yr_srtm1}\n"
printf "dbg: yr_endm1 = ${yr_endm1}\n"
if [ ${mpi_flg} = 'Yes' ]; then
for ((nd_idx=0;nd_idx<${nd_nbr};nd_idx++)); do
printf "dbg: nd_nm[${nd_idx}] = ${nd_nm[${nd_idx}]}\n"
done # !nd
fi # !mpi
fi # !dbg
if [ ${dbg_lvl} -ge 2 ]; then
psn_nbr=$#
printf "dbg: Found ${psn_nbr} positional parameters (besides \$0):\n"
for ((psn_idx=1;psn_idx<=psn_nbr;psn_idx++)); do
printf "dbg: psn_arg[${psn_idx}] = ${!psn_idx}\n"
done # !psn_idx
fi # !dbg
if [ ${dbg_lvl} -ge 2 ]; then
if [ ${inp_aut} = 'Yes' ] || [ ${inp_glb} = 'Yes' ] || [ ${inp_std} = 'Yes' ]; then
printf "dbg: Obtained or generated ${fl_nbr} input files:\n"
for ((fl_idx=0;fl_idx<fl_nbr;fl_idx++)); do
printf "dbg: fl_in[${fl_idx}] = ${fl_in[${fl_idx}]}\n"
done # !fl_idx
fi # !inp_aut
fi # !dbg
# Human-readable summary
date_srt=$(date +"%s")
if [ ${dbg_lvl} -ge 0 ]; then
printf "Climatology operations invoked with command:\n"
echo "${cmd_ln}"
fi # !dbg
if [ ${dbg_lvl} -ge 2 ]; then
printf "************************************************************************\n"
printf "IMPORTANT: This output shows a SIMULATION of a production run\n"
printf "The values shown are real, but the printed commands are NOT executed\n"
printf "This output is for debugging purpose only since dbg_lvl = ${dbg_lvl} >= 2\n"
printf "The elapsed time should be near 0m0s since there is very little disk I/O\n"
printf "Set dbg_lvl <= 1 for a PRODUCTION RUN that executes these commands\n"
printf "************************************************************************\n"
fi # !dbg
if [ "${tms_flg}" = 'Yes' ]; then
printf "Started climatology splitting at `date`\n"
else
printf "Started climatology generation at `date`\n"
fi # !tms_flg
printf "Running climatology script ${spt_nm} from directory ${drc_spt}\n"
printf "NCO binaries version ${nco_vrs} from directory ${drc_nco}\n"
printf "Parallelism mode for files = ${par_typ}\n"
if [ "${tms_flg}" = 'Yes' ]; then
if [ ${var_nbr} -gt 1 ]; then
printf "Timeseries will be created for each of ${var_nbr} variables\n"
if [ "${par_typ}" = ${par_bck} ]; then
printf "Background parallelism processing variables in var_nbr/job_nbr = ${var_nbr}/${job_nbr} = $((var_nbr / job_nbr)) sequential batches each concurrently processing job_nbr = ${job_nbr} jobs (1 per variable), then remaining $((var_nbr % job_nbr)) jobs/variables simultaneously\n"
elif [ "${par_typ}" = ${par_mpi} ]; then
printf "MPI parallelism dividing processing of ${var_nbr} variables onto nd_nbr = ${nd_nbr} nodes simultaneously\n"
fi # !par_typ
if [ "${par_typ}" = ${par_bck} ] || [ "${par_typ}" = ${par_mpi} ]; then
if [ "${job_nbr}" -gt ${job_nbr_wrn} ] && [ ${var_nbr} -gt ${job_nbr_wrn} ]; then
printf "WARNING: Requested number of simultaneous jobs = job_nbr = ${job_nbr} exceeds threshold number = job_nbr_wrn = ${job_nbr_wrn}. This command will start an unusually (and possibly inadvertently) large number of splitter tasks for most computers. Consequences may include insufficient RAM that leads to swapping, slow performance due to I/O contention when reading/writing data.\n"
printf "HINT: If undesirable performance occurs, use the --job_nbr option to reduce the number of simultaneous jobs, e.g., ${spt_nm} --job_nbr=100 ...\n"
fi # !job_nbr
fi # !par_typ
else # !var_nbr
printf "Timeseries will be created for only one variable\n"
fi # !var_nbr
rmd_units='year'
if [ "${mth_flg}" = 'Yes' ]; then
rmd_units='month'
fi # !mth_flg
sgm_sng='0'
if [ ${sgm_rmd} -ne 0 ]; then
sgm_sng='1'
fi # !sgm_rmd
if [ ${sgm_nbr} -gt 1 ]; then
printf "All this occurs within an outer loop of (yr_sbs/ypf_max) + (remainder, if any) = ${yr_sbs}/${ypf_max} + ${sgm_sng} = ${sgm_nbr} time segments\n"
if [ ${rmd_flg} = 'Yes' ]; then
printf "Will split data for each variable into ${sgm_nbrm1} timeseries segment(s) of length ${ypf_max} years and 1 segment of length ${sgm_rmd} ${rmd_units}(s)\n"
else # !rmd_flg
printf "Will split data for each variable into ${sgm_nbr} timeseries segment(s) of length ${ypf_max} years\n"
fi # !rmd_flg
else # !sgm_nbr
if [ ${mth_nbr} -eq 12 ]; then
printf "Will split data for each variable into one timeseries of length ${yr_sbs} years\n"
else
if [ ${clm_md} = 'hfs' ]; then
printf "Will split data for each variable into one timeseries of length ${yr_sbs} years\n"
else
if [ "${mth_flg}" = 'Yes' ]; then
printf "Will split data for each variable into one timeseries of length ${yr_sbs} years and ${sgm_rmd} months\n"
else
printf "Will split data for each variable into one timeseries of length ${yr_sbs} years and ${fl_rmd} months\n"
fi # !mth_flg
fi # !clm_md
fi # !mth_nbr
fi # !sgm_nbr
if [ ${inp_std} = 'No' ]; then
if [ "${drc_in_usr_flg}" = 'Yes' ]; then
printf "Splitting climatology from ${fl_nbr} raw input files in directory ${drc_in}\n"
else # !drc_in
printf "Splitting climatology from ${fl_nbr} raw input files specified as positional arguments\n"
fi # !drc_in
else
printf "Splitting climatology from list of ${fl_nbr} raw input files piped to stdin\n"
fi # !stdin
if [ ${clm_md} = 'ann' ]; then
printf "Each input file assumed to contain statistics for one year\n"
elif [ ${clm_md} = 'dly' ]; then
printf "Each input file assumed to contain statistics for one day\n"
elif [ ${clm_md} = 'hfs' ]; then
printf "Each input file assumed to contain statistics for ${dpf} days each comprising ${tpd_out} timesteps\n"
elif [ ${clm_md} = 'mth' ]; then
printf "Each input file assumed to contain statistics for one month\n"
fi # !mth
if [ "${rgn_stt}" != 'No' ]; then
if [ "${rgn_stt}" = 'avg' ]; then
printf "Hemispherically and globally averaged timeseries files to be saved to directory ${drc_out}\n"
elif [ "${rgn_stt}" = 'sum' ]; then
printf "Hemispherically and globally integrated (summed) timeseries files to be saved to directory ${drc_out}\n"
fi # !rgn_stt
else
printf "Native-grid timeseries files to be saved to directory ${drc_out}\n"
fi # !rgn_stt
if [ -n "${rgr_opt}" ]; then
printf "Regridded split files to directory ${drc_rgr}\n"
if [ "${thr_nbr}" -ne 1 ]; then
printf "Regridder will request ${thr_nbr} OpenMP threads to parallelize weight application across variables\n"
fi # !thr_nbr
else
printf "Split files will not be regridded\n"
fi # !rgr
fi # !tms_flg
if [ "${clm_flg}" = 'Yes' ]; then
if [ "${clm_hfc_or_mth}" = 'Yes' ] && [ "${bnr_flg}" = 'No' ]; then
if [ "${par_typ}" = ${par_bck} ]; then
printf "Background parallelism processing monthly climos in 12/job_nbr = 12/${job_nbr} = $((12 / job_nbr)) sequential batches each handling job_nbr = ${job_nbr} months concurrently\n"
elif [ "${par_typ}" = ${par_mpi} ]; then
printf "MPI parallelism processing monthly climos in 12/job_nbr = 12/${job_nbr} = $((12 / job_nbr)) sequential batches. Each batch is distributed to ${nd_nbr} nodes via round-robin scheduling and handles job_nbr = ${job_nbr} months concurrently. Load-balancing occurs with unused nodes, if any.\n"
fi # !par_typ
fi # !clm_md
if [ "${xtn_flg}" = 'No' ]; then
printf "Producing standard climatology from raw input files in directory ${drc_in}\n"
printf "Output files to directory ${drc_out}\n"
fi # !xtn_flg
if [ "${bnr_flg}" = 'Yes' ]; then
printf "Producing extended climatology in binary mode: Will combine pre-computed climatology in directory ${drc_prv} with pre-computed climatology in directory ${drc_in}\n"
printf "Output files to directory ${drc_xtn}\n"
fi # !bnr_flg
if [ "${ncr_flg}" = 'Yes' ]; then
printf "Producing extended climatology in incremental mode: Pre-computed climatology in directory ${drc_prv} will be incremented by raw input files in directory ${drc_in}\n"
printf "Output files to directory ${drc_xtn}\n"
fi # !ncr_flg
if [ "${bnr_flg}" = 'No' ]; then
printf "Climatology from ${yr_nbr} years of contiguous raw data touching ${yr_cln} calendar years from YYYYMM = ${yyyy_clm_srt_dec}${mm_ann_srt} to ${yyyy_end}${mm_ann_end}\n"
fi # !bnr_flg
if [ "${inp_aut}" = 'Yes' ]; then
if [ "${nm_typ}" = 'yyyymm' ]; then
printf "Input ilenames will be constructed with generic conventions as ${bs_nm}_YYYYMM.${bs_sfx}\n"
elif [ "${nm_typ}" = 'yyyy-mm' ]; then
printf "Input filenames will be constructed with generic conventions as ${bs_nm}.YYYY-MM.${bs_sfx}\n"
elif [ "${nm_typ}" = 'yyyy-mm-01' ]; then
printf "Input filenames will be constructed with generic conventions as ${bs_nm}.YYYY-MM-01.${bs_sfx}\n"
elif [ "${nm_typ}" = 'yyyy-mm-01-00000' ]; then
printf "Input filenames will be constructed with generic conventions as ${bs_nm}.YYYY-MM-01-00000.${bs_sfx}\n"
elif [ "${nm_typ}" = 'mpas' ]; then
printf "Input filenames will be constructed with MPAS conventions\n"
else
printf "Input filenames will be constructed with CESM'ish or E3SM'ish conventions\n"
fi # !nm_typ
else
printf "Input filenames will be read (not constructed) from globbing, positional arguments, or stdin\n"
fi # !inp_aut
if [ ${clm_md} = 'ann' ]; then
printf "Each input file assumed to contain statistic for one year\n"
elif [ ${clm_md} = 'dly' ]; then
printf "Each input file assumed to contain statistics for one or more daily or sub-daily (e.g., 3-hourly) means\n"
elif [ ${clm_md} = 'hfc' ]; then
printf "Each input file assumed to contain statistics for an integer number of days each comprising ${tpd_out} timesteps\n"
elif [ ${clm_md} = 'mth' ]; then
printf "Each input file assumed to contain statistic for one month\n"
fi # !mth
if [ "${clm_hfc_or_mth}" = 'Yes' ]; then
if [ ${csn_flg} = 'No' ]; then
printf "Seasons and therefore annual means derived from seasonal means have been turned-off and will not be computed\n"
elif [ ${wnt_md} = 'djf' ]; then
printf "Winter statistics based on seasonally contiguous December (djf-mode): DJF-means formed from consecutive DJF months that cross calendar-year boundaries\n"
else
printf "Winter statistics based on seasonally discontiguous December (jfd-mode): Means labeled as \"DJF\" are formed from JFD months from the same calendar year\n"
fi # !djf
fi # !mth
if [ ${cf_flg} = 'Yes' ]; then
printf "Annotation for CF climatology attribute and climatology_bounds variable will be performed\n"
else
printf "Annotation for CF climatology attribute and climatology_bounds variable will not be performed\n"
fi # !cf
if [ -n "${rgr_opt}" ]; then
printf "This climatology will also be regridded\n"
printf "Regridder will request ${thr_nbr} OpenMP threads to parallelize weight application across variables\n"
else
printf "This climatology will not be regridded\n"
fi # !rgr
fi # !clm_flg
# Block 1: Generate, check, and store (but do not yet execute) commands
# Block 1 Loop 1: Produce monthly-mean climatology output
if [ "${clm_flg}" = 'Yes' ] && [ "${clm_hfc_or_mth}" = 'Yes' ]; then
clm_idx=0
for mth in {01..12}; do
let clm_idx=${clm_idx}+1
MM=`printf "%02d" ${clm_idx}`
# Monthly filenames are every twelfth file from input list (which is assumed to start with January)
if [ ${clm_md} = 'mth' ]; then
fl_all=''
for ((yr_idx=0;yr_idx<yr_nbr;yr_idx++)); do
if [ ${wnt_md} = 'jfd' ]; then
let fl_idx=$((yr_idx * 12 + clm_idx - 1))
else
if [ ${MM} != '12' ]; then
let fl_idx=$((yr_idx * 12 + clm_idx))
else # !dec
let fl_idx=$(((yr_idx - 1) * 12 + clm_idx))
fi # !dec
fi # !wnt_md
fl_all="${fl_all} ${fl_in[${fl_idx}]}"
done # !yr_idx
# Check for existence of raw input only when file will be used
if [ "${bnr_flg}" = 'No' ]; then
for fl_crr in ${fl_all} ; do
if [ ! -f "${drc_in}/${fl_crr}" ] && [ ! -L "${drc_in}/${fl_crr}" ]; then
echo "${spt_nm}: ERROR Unable to find required input file ${drc_in}/${fl_crr}"
echo "${spt_nm}: HINT All files implied to exist by the climatology bounds (start/end year/month) and by the specified (with -P or -m) or default model type, must be in ${drc_in} before ${spt_nm} will proceed"
exit 1
fi # ! -f
done # !fl_crr
else # !bnr_flg
# In binary mode drc_out is actually used to locate input files from climatology B (same as output files in incremental mode)
drc_out="${drc_in}"
fi # !bnr_flg
fi # !clm_md
# Common to both high-frequency and monthly-input climos
if [ ${wnt_md} = 'djf' ] && [ ${MM} = '12' ]; then
yyyy_clm_srt=${yyyy_clm_srt_dec}
yyyy_clm_end=${yyyy_clm_end_dec}
fi # !djf
fl_out[${clm_idx}]="${drc_out}/${out_nm}_${MM}_${yyyy_clm_srt}${MM}_${yyyy_clm_end}${MM}_climo.nc"
if [ ${clm_md} = 'hfc' ]; then
# Computing climos from high-freqency data requires many steps:
# A primary goal is to re-use not re-write code, especially existing monthly frequency climo code
# Otherwise code maintainability significantly suffers
# Code-reuse all-but-requires same existing parallelization as monthly frequency climos
# Difficulty is that traditionaly monthly mean climos equally weight all input records using ncra
# Proposed Algorithm:
# 1. Re-use splitter option ypf for intermediate files so users can prevent excessive intermediate filesizes
# 2. Generate month-aligned intermediate segments containing ypf instances of a single month (and no other months) for all variables
# 2a. To subset repeated appearances of given month and skip intervening 11 months, command must have form ncrcat -d time,srt_sng,end_sng,srd,ssc where srd=dpy*tpd and ssc=dpm*tpd
# 2b. Same as draft annual-mean HFS feature for diurnal cycle, except srt_sng/end_sng are "yyyy_srt_sgm-MM-01 00:00:01/yyyy_end_sgm-(MMp1)-01 00:00:00" instead of "yyyy_srt_sgm-01-01 00:00:01/yyyy_endp1_sgm-01-01 00:00:00"
# 2c. Intermediate monthly segments sgm_yyyy_srt_yyyy_end_MM.nc have ypf instances of given month, except last segment may be ragged
# 2d. Evaluate mean diurnal cycle of monthly-segment using ncra --d time,srt,end,srd=ssc,ssc=dpm*tpd,tpd
# 2c. Ragged numbers of ypf in last segment suggests using ncra climatological monthly diurnal-mean output
# 2d. Parallelize this over month (probably) using job_nbr algorithm? or over segment (unlikely)?
# 2e. Parallelization is difficult/tedious due to number of basic blocks (3? 4? 5?)
# 3. Intermediate files of shape (time=min(actual,ypf),hour,space) are nearly interchangeable with monthly-frequency output in stage-1 ncra
# 4. Use climatological monthly diurnal-mean output to create climatological seasonal diurnal-mean output
# 5. Use climatological monthly seasonal-mean output to create climatological annual diurnal-mean output
# 6. Final output climo format: (time=1,hour,space) (or convert to (time=hour,space)?)
# Advantages:
# 1. Code-reuse and maintainability
# 2. Scales to any length o simulation (intermediate files can be a single month)
# 3. Output similar to existing monthly-frequency climos with new, fixed-length 'hour' dimension
# 4. Climatology bounds attributes correct for time variable(?)
# Disadvantages:
# 1. E3SM-diags post-processing may not like new 'hour' dimension
# 2. No corresponding examples in CF Conventions
# 3. Any utility in multiple-variables per output file, e.g., TS, PRECT? Would single variables suffice?
DD=`printf "%02d" ${dpm[${clm_idx}]}`
tm_srt="${yyyy_clm_srt}-${MM}-01T00:00:01"
let mthp1=$((clm_idx + 1))
MMp1=`printf "%02d" ${mthp1}`
tm_end="${yyyy_clm_end}-${MMp1}-01T00:00:00"
yyyy_clm_endp1=`printf "%04d" ${yr_endp1}`
if [ ${clm_idx} -eq 12 ]; then
tm_srt="${yyyy_clm_srt_dec}-${MM}-01T00:00:01"
yyyy_clm_endp1=`printf "%04d" ${yr_endp1}`
if [ ${wnt_md} = 'djf' ]; then
yyyy_clm_endp1=`printf "%04d" ${yr_end}`
fi # !djf
tm_end="${yyyy_clm_endp1}-01-01T00:00:00"
fi # !December
let srd=${dpy}*${tpd_out}
let ssc=${dpm[${clm_idx}]}*${tpd_out}
# Cull this month from interannual climatology into interannual diurnal file
fl_mth[${clm_idx}]="${drc_out}/${out_nm}_${MM}_tms_foo_${yyyy_clm_srt}_${yyyy_clm_end}.nc"
# Command to produce interleaved monthly climos
if [ "${fl_nbr}" -eq 1 ]; then
# Compute interleaved mean of single-file input (e.g., CMIP timeseries format)
#cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra -O ${nco_opt} ${gaa_sng} -d time,${tm_srt},${tm_end},${srd},${ssc},${tpd_out} ${fl_in} ${fl_out[${clm_idx}]}"
tm_nbr=`ncks --trd -M ${ppn_opt} ${fl_in} | grep -E -i "^Root record dimension 0:" | cut -f 10- -d ' '`
let ssc2=${tm_nbr}*${tpd_out}/${srd}
# Ensure result is positive integer
[ "${ssc2}" -ge 0 ] 2>/dev/null && ssc2_ntg='Yes' || ssc2_ntg='No'
if [ "${ssc2_ntg}" != 'Yes' ]; then
printf "${spt_nm}: ERROR Inferred value of ssc2=${ssc2} is not a positive integer. User requested high-frequency climatology with one input file. ${spt_nm} attempted to divine the sub-cycle length based on the assumption that the file contains and integral number of years (at any temporal resolution) of input data. The divination resulted in a non-integral sub-cycle length. This is not surprising since a number of assumptions must hold true for the inferral to work correctly.\nHINT: Clip the input timeseries to an integral number of years.\n"
exit 1
fi # !ssc2_ntg
cmd_cll[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra -O ${nco_opt} ${gaa_sng} -d time,${tm_srt},${tm_end},${srd},${ssc},${tpd_out} ${fl_in} ${fl_mth[${clm_idx}]}"
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra --clm_bnd=${yr_srt},${yr_end},${clm_idx},${clm_idx},${tpd_out} -O ${nco_opt} ${gaa_sng} -d time,,,,${ssc2},${tpd_out} ${fl_mth[${clm_idx}]} ${fl_out[${clm_idx}]}"
else # !fl_nbr
# Concatenate means of each interleaved index of multi-file input (e.g., raw history output or multiple segments)
cmd_cll[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncrcat -O ${nco_opt} ${gaa_sng} -d time,${tm_srt},${tm_end},${srd},${ssc} ${ppn_opt} ${fl_in[*]} ${fl_mth[${clm_idx}]}"
# Take mean value of each diurnal timestep across all instances of this month
fl_tpd_tpl[${clm_idx}]="${drc_out}/${out_nm}_${MM}_tpdTPDTT_${yyyy_clm_srt}_${yyyy_clm_end}.nc"
cmd_tpd_tpl[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra -O ${nco_opt} ${gaa_sng} -d time,TPDidx,,${tpd_out} ${fl_mth[${clm_idx}]} ${fl_tpd_tpl[${clm_idx}]}"
# Concatenate individual timestep means into full diurnal timeseries
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncrcat -O --clm_bnd=${yr_srt},${yr_end},${clm_idx},${clm_idx},${tpd_out} ${nco_opt} ${gaa_sng} ${drc_out}/${out_nm}_${MM}_tpd* ${fl_out[${clm_idx}]}"
fi # !fl_nbr
# Combine monthly into seasonal and annual means with nces (preserving diurnal timesteps)
cmd_mth2csn='nces'
cmd_csn2ann='nces'
elif [ ${clm_md} = 'mth' ]; then
clm_bnd[${clm_idx}]=${yyyy_clm_srt},${yyyy_clm_end},${clm_idx},${clm_idx},0
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra --clm_bnd=${clm_bnd[${clm_idx}]} -O ${nco_opt} ${gaa_sng} ${ppn_opt} ${fl_all} ${fl_out[${clm_idx}]}"
# cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra --clm_bnd=${yr_srt},${yr_end},${clm_idx},${clm_idx},0 -O ${nco_opt} ${gaa_sng} ${ppn_opt} ${fl_all} ${fl_out[${clm_idx}]}"
# Combine monthly into seasonal and annual means with ncra
cmd_mth2csn='ncra'
cmd_csn2ann='ncra'
fi # !clm_md
done # !mth
if [ ${clm_md} = 'hfc' ]; then
if [ "${fl_nbr}" -gt 1 ]; then
printf "Cull each month from interannual climatology into interannual diurnal file...\n"
else
printf "Create interannual timeseries of monthly diurnal timestep means from single file...\n"
fi # !fl_nbr
[[ ${dbg_lvl} -ge 1 ]] && date_cll=$(date +"%s")
if [ ${job_nbr} -eq 1 ] || [ ${job_nbr} -eq 2 ] || [ ${job_nbr} -eq 3 ] || [ ${job_nbr} -eq 4 ] || [ ${job_nbr} -eq 6 ] || [ ${job_nbr} -eq 12 ]; then
echo "Successful match of job_nbr to list of permissible values" > /dev/null
else
echo "${spt_nm}: ERROR Job number job_nbr=${job_nbr} is invalid in monthly climo mode. Must be a factor of 12. Valid values are 1, 2, 3, 4, 6, and 12. Please re-submit with valid job_nbr."
exit 1
fi # !job_nbr
# Batch number is inverse to job number, so job_nbr=1->bch_nbr=12 (serial mode, slowest), job_nbr=2->bch_nbr=6, ... job_nbr=12->bch_nbr=1 (full background mode, fastest)
let bch_nbr=$((12 / job_nbr))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# clm_idx is 1-based, bch_idx is 0-based
let clm_idx_srt=$(((bch_idx * job_nbr) + 1))
let clm_idx_end=$((clm_idx_srt + job_nbr - 1))
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
printf "Cull climatological monthly timeseries for month ${clm_idx} ...\n"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_cll[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_cll[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly cull cmd_cll[${clm_idx}] failed. Debug this:\n${cmd_cll[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_cll[${clm_idx}]} ${par_opt} # eval always returns 0 on backgrounded processes
cll_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
if [ -n "${par_opt}" ]; then
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
wait ${cll_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly cull cmd_cll[${clm_idx}] failed. Debug this:\n${cmd_cll[${clm_idx}]}\n"
# 20200805: exiting parent here creates orphans, kill live children then exit
for ((kid_idx=clm_idx+1;kid_idx<=clm_idx_end;kid_idx++)); do
kill -9 ${cll_pid[${kid_idx}]}
done # !kid_idx
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
done # !bch_idx
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_cll))
echo "Elapsed time to cull monthly timeseries $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
if [ "${fl_nbr}" -gt 1 ]; then
printf "Take mean value of each diurnal timestep across all instances of every month...\n"
[[ ${dbg_lvl} -ge 1 ]] && date_tpd=$(date +"%s")
for ((tpd_idx=0;tpd_idx<tpd_out;tpd_idx++)); do
TT=`printf "%02d" ${tpd_idx}`
printf "Compute monthly means for diurnal timestep ${tpd_idx}...\n"
let bch_nbr=$((12 / job_nbr))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# clm_idx is 1-based, bch_idx is 0-based
let clm_idx_srt=$(((bch_idx * job_nbr) + 1))
let clm_idx_end=$((clm_idx_srt + job_nbr - 1))
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
printf "De-interleave and average diurnal timestep ${tpd_idx} for month ${clm_idx} ...\n"
cmd_tpd[${clm_idx}]=${cmd_tpd_tpl[${clm_idx}]/TPDTT/${TT}}
cmd_tpd[${clm_idx}]=${cmd_tpd[${clm_idx}]/TPDidx/${tpd_idx}}
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_tpd[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_tpd[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly de-interleave and average command cmd_tpd[${clm_idx}] failed. Debug this:\n${cmd_tpd[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_tpd[${clm_idx}]} ${par_opt} # eval always returns 0 on backgrounded processes
tpd_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
if [ -n "${par_opt}" ]; then
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
wait ${tpd_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly de-interleave and average command cmd_tpd[${clm_idx}] failed. Debug this:\n${cmd_tpd[${clm_idx}]}\n"
# 20200805: exiting parent here creates orphans, kill live children then exit
for ((kid_idx=clm_idx+1;kid_idx<=clm_idx_end;kid_idx++)); do
kill -9 ${tpd_pid[${kid_idx}]}
done # !kid_idx
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
done # !bch_idx
done # !tpd_idx
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_tpd))
echo "Elapsed time to de-interleave and average all timesteps $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
# Clean-up monthly inputs (important since these file can be huge)
for ((clm_idx=1;clm_idx<=12;clm_idx++)); do
/bin/rm -f ${fl_mth[${clm_idx}]}
done # !clm_idx
fi # !fl_nbr
fi # !clm_md
if [ ${clm_md} = 'mth' ]; then
tpd_out=0
fi # !tpd_out
# Monthly output filenames constructed above; specify remaining (seasonal, annual) output names
for ((csn_idx=0;csn_idx<${csn_nbr};csn_idx++)); do
# Produces names like "${out_nm}_MAM_${yyyy_srt}03_${yyyy_end}05_climo.nc"
let clm_idx=1+12+${csn_idx}
fl_out[${clm_idx}]="${drc_out}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_srt}${csn_srt_mm[${map_r2d[${csn_idx}]}]}_${yyyy_end}${csn_end_mm[${map_r2d[${csn_idx}]}]}_climo.nc"
clm_bnd[${clm_idx}]="--clm_bnd=${yr_srt},${yr_end},${csn_srt_idx[${map_r2d[${csn_idx}]}]},${csn_end_idx[${map_r2d[${csn_idx}]}]},${tpd_out}"
if [ "${csn_abb[${map_r2d[${csn_idx}]}]}" = 'DJF' ]; then
fl_out[${clm_idx}]="${drc_out}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec}${mm_djf_srt}_${yyyy_end}${mm_djf_end}_climo.nc"
fl_out[${clm_idx}]="${drc_out}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec}${mm_djf_srt}_${yyyy_end}${mm_djf_end}_climo.nc"
clm_bnd[${clm_idx}]="--clm_bnd=${yyyy_clm_srt_dec},${yyyy_clm_end_dec},${mm_djf_srt},${mm_djf_end},${tpd_out}"
fi # !DJF
if [ "${csn_abb[${map_r2d[${csn_idx}]}]}" = 'ANN' ]; then
fl_out[${clm_idx}]="${drc_out}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec}${mm_ann_srt}_${yyyy_end}${mm_ann_end}_climo.nc"
fl_out[${clm_idx}]="${drc_out}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec}${mm_ann_srt}_${yyyy_end}${mm_ann_end}_climo.nc"
clm_bnd[${clm_idx}]="--clm_bnd=${yyyy_clm_srt_dec},${yyyy_end},${mm_ann_srt},${mm_ann_end},${tpd_out}"
fi # !ANN
done # !csn_idx
# fl_out[13]="${drc_out}/${out_nm}_MAM_${yyyy_srt}03_${yyyy_end}05_climo.nc"
# fl_out[14]="${drc_out}/${out_nm}_JJA_${yyyy_srt}06_${yyyy_end}08_climo.nc"
# fl_out[15]="${drc_out}/${out_nm}_SON_${yyyy_srt}09_${yyyy_end}11_climo.nc"
# fl_out[16]="${drc_out}/${out_nm}_DJF_${yyyy_clm_srt_dec}${mm_djf_srt}_${yyyy_end}${mm_djf_end}_climo.nc"
if [ "${ann_flg}" = 'Yes' ]; then
fl_out[${clm_nbr}]="${drc_out}/${out_nm}_ANN_${yyyy_clm_srt_dec}${mm_ann_srt}_${yyyy_end}${mm_ann_end}_climo.nc"
clm_bnd[${clm_nbr}]="--clm_bnd=${yyyy_clm_srt_dec},${yyyy_end},${mm_ann_srt},${mm_ann_end},${tpd_out}"
fi # !ann_flg
# Derive all regridded and AMWG names from output names
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
fl_amwg[${clm_idx}]=`expr match "${fl_out[${clm_idx}]}" '\(.*\)_.*_.*_climo.nc'` # Prune _YYYYYMM_YYYYMM_climo.nc
fl_amwg[${clm_idx}]="${fl_amwg[${clm_idx}]}_climo.nc" # Replace with _climo.nc
fl_amwg[${clm_idx}]="${fl_amwg[${clm_idx}]/${drc_out}\//}" # Delete prepended path to ease symlinking
if [ -n "${rgr_opt}" ]; then
fl_rgr[${clm_idx}]="${fl_out[${clm_idx}]/${drc_out}/${drc_rgr}}"
if [ "${drc_out}" = "${drc_rgr}" ]; then
# Append geometry suffix to regridded files in same directory as native climo
# http://tldp.org/LDP/abs/html/string-manipulation.html
dfl_sfx='rgr'
rgr_sfx=`expr match "${rgr_map}" '.*_to_\(.*\).nc'`
if [ "${#rgr_sfx}" -eq 0 ]; then
printf "${spt_nm}: WARNING Unable to extract geometric suffix from mapfile, will suffix regridded files with \"${dfl_sfx}\" instead\n"
rgr_sfx=${dfl_sfx}
else
yyyymmdd_sng=`expr match "${rgr_sfx}" '.*\(\.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]\)'` # Find YYYYYMMDD
if [ "${#yyyymmdd_sng}" -ne 0 ]; then
rgr_sfx=${rgr_sfx%%${yyyymmdd_sng}} # Delete YYYYYMMDD
fi # !strlen
fi # !strlen
# rgr_sfx=`expr match "${rgr_sfx}" '\(.*\)\.[0-9][0-9][0-9][0-9][0-9][0-9]'` #
fl_rgr[${clm_idx}]="${fl_rgr[${clm_idx}]/.nc/_${rgr_sfx}.nc}"
fi # !drc_rgr
fi # !rgr_opt
done # !clm_idx
fi # !clm_md
# Block 1 Loop N: Split
if [ "${tms_flg}" = 'Yes' ]; then
# Populate input file arguments fl_sgm with all files necessary to bracket each segment
for ((sgm_idx=0;sgm_idx<sgm_nbr;sgm_idx++)); do
fl_sgm[${sgm_idx}]=''
if [ "${clm_md}" != 'hfs' ]; then
# Non-high-frequency timeseries receive exact file-lists to hyperslab by index (not by time-coordinate)
if [ "${mth_flg}" = 'Yes' ]; then
let fl_idx_srt=${sgm_idx}*${ypf_max}*${fpy} # NB: 0-based
let fl_idx_end=${fl_idx_srt}+${ypf_max}*${fpy} # NB: 1-based
let yr_srt_sgm=${yr_srt_rth}+${sgm_idx}*${ypf_max}
let yr_end_sgm=${yr_srt_sgm}+${ypf_max}
if [ ${sgm_rmd} -ne 0 ] && [ ${sgm_idx} -eq ${sgm_nbrm1} ]; then
let fl_idx_end=${fl_idx_srt}+${sgm_rmd} # NB: 1-based
let yr_end_sgm=${yr_end_rth}
fi # !sgm_nbrm1
else # !mth_flg
let fl_idx_srt=${sgm_idx}*${ypf_max}*${fpy} # NB: 0-based
let fl_idx_end=${fl_idx_srt}+${ypf_max}*${fpy} # NB: 1-based
let yr_srt_sgm=${yr_srt_rth}+${sgm_idx}*${ypf_max}
let yr_end_sgm=${yr_srt_sgm}+${ypf_max}-1
if [ ${sgm_rmd} -ne 0 ] && [ ${sgm_idx} -eq ${sgm_nbrm1} ]; then
let fl_idx_end=${fl_idx_srt}+${sgm_rmd}*${fpy} # NB: 1-based
let yr_end_sgm=${yr_srt_sgm}+${sgm_rmd}-1
fi # !sgm_nbrm1
fi # !mth_flg
else # !clm_md
# High-frequency timeseries are hyperslabbed by date and so need fuzzy file-lists
# that may contain an extra file or two in case year-boundaries are intermediate records.
# 20210331: Prior to now, output high-frequency timeseries always start/end on year boundaries
# 20210331: Support high-frequency splitter mode mth_srt/end != Jan/Dec
# However, the number of files to read in each segment may differ, and is not known a priori
# NB: High-frequency timeseries underestimate fpy by [0,1) when fpy=dpy/dpf is non-integral
# 20210329: Current formulation assumes manual minimal file list is specified
# Extraneous (more than one non-overlapping) files at beginning of list will cause failure
# fxm: Omit extraneous files at start of list (extraneous at end are OK now)
let fpyp1=${fpy}+1
# 20220818: User Jinbo Xie has hfs files with dpf=36.5. Bash chokes on FP arithmetic
let fl_idx_srt=${sgm_idx}*${ypf_max}*${dpy}/${dpf} # NB: dpy/dpf more accurate than fpy
#fl_idx_srt=$(echo "${sgm_idx}*${ypf_max}*${dpy}/${dpf}" | bc -l) # NB: dpy/dpf more accurate than fpy
#echo "dbg: fl_idx_srt = ${fl_idx_srt}"
let fl_idx_end=${fl_idx_srt}+${ypf_max}*${fpyp1} # NB: use fpyp1 to overestimate fl_idx_end
#echo "dbg: fl_idx_end = ${fl_idx_end}"
let yr_srt_sgm=${yr_srt_rth}+${sgm_idx}*${ypf_max}
let yr_end_sgm=${yr_srt_sgm}+${ypf_max}-1
if [ ${sgm_rmd} -ne 0 ] && [ ${sgm_idx} -eq ${sgm_nbrm1} ]; then
if [ "${mth_flg}" = 'Yes' ]; then
# 20210331 Peings
let fl_idx_end=${fl_idx_srt}+${sgm_rmd}
let yr_end_sgm=${yr_end_rth}
else # !mth_flg
let fl_idx_end=${fl_idx_srt}+${sgm_rmd}*${fpyp1} # NB: use fpyp1 to overestimate fl_idx_end
let yr_end_sgm=${yr_srt_sgm}+${sgm_rmd}-1
fi # !mth_flg
fi # !sgm_rmd || !sgm_nbrm1
# Fuzzify list by including files nominally before and after first and last expected files
if [ ${fl_idx_srt} -ne 0 ]; then
let fl_idx_srt=${fl_idx_srt}-1
fi # !fl_idx_srt
# Fuzziness may generate indices that are too large
if [ ${fl_idx_end} -gt ${fl_nbr} ]; then
let fl_idx_end=${fl_nbr}
fi # !fl_idx_end
if [ ${fl_idx_end} -lt ${fl_nbr} ]; then
let fl_idx_end=${fl_idx_end}+1
fi # !fl_idx_end
fi # !clm_md
# NB: fl_idx_srt is zero-based while fl_idx_end one-based, i.e., the count, one more than the index
for ((fl_idx=fl_idx_srt;fl_idx<fl_idx_end;fl_idx++)); do
fl_sgm[${sgm_idx}]="${fl_sgm[${sgm_idx}]} ${fl_in[${fl_idx}]}"
done # !fl_idx
yyyy_srt_sgm[${sgm_idx}]=`printf "%04d" ${yr_srt_sgm}`
yyyy_end_sgm[${sgm_idx}]=`printf "%04d" ${yr_end_sgm}`
# 202401001: fxm: presumably could cause value too great for base error?
let yr_endp1_sgm=${yr_end_sgm}+1
yyyy_endp1_sgm[${sgm_idx}]=`printf "%04d" ${yr_endp1_sgm}`
mm_srt_sgm[${sgm_idx}]='01'
mm_end_sgm[${sgm_idx}]='12'
mm_srt_sgm[${sgm_idx}]=${mm_srt}
if [ ${sgm_idx} -eq ${sgm_nbrm1} ]; then
let mth_end_sgm=${mth_end_rth}
mm_end_sgm[${sgm_idx}]=${mm_end}
else # !sgm_idx
let mth_end_sgm=${mth_srtm1}
mm_end_sgm[${sgm_idx}]=${mm_srtm1}
fi # !sgm_idx
# 202401001: fxm: Causes error "value too great for base (error token is "08")"
let mm_endp1_sgm=${mth_end_sgm}+1
mm_endp1_sgm[${sgm_idx}]=`printf "%02d" ${mm_endp1_sgm}`
# Sanity check that all files exist before wasting time and space on concatenation
for fl_crr in ${fl_sgm[${sgm_idx}]} ; do
if [ ! -f "${drc_in}/${fl_crr}" ] && [ ! -L "${drc_in}/${fl_crr}" ]; then
echo "${spt_nm}: ERROR Unable to find required input file ${drc_in}/${fl_crr}"
echo "${spt_nm}: HINT All files implied to exist by the climatology bounds (start/end year/month) and by the specified (with -P or -m) or default model type, must be in ${drc_in} before ${spt_nm} will proceed"
exit 1
fi # ! -f
done # !fl_crr
done # !sgm_idx
if [ "${clm_md}" = 'hfs' ]; then
if [ ${dbg_lvl} -ge 1 ]; then
printf "dbg: HFS variable(s):\n"
printf "dpf = ${dpf}\n"
printf "dpy = ${dpy}\n"
printf "fl_idx_srt = ${fl_idx_srt}\n"
printf "fl_idx_end = ${fl_idx_end}\n"
printf "fl_nbr = ${fl_nbr}\n"
printf "fpy = ${fpy}\n"
printf "fpyp1 = ${fpyp1}\n"
printf "nco_opt = ${nco_opt}\n"
printf "ppn_opt = ${ppn_opt}\n"
printf "sgm_nbr = ${sgm_nbr}\n"
printf "sgm_rmd = ${sgm_rmd}\n"
printf "spl_opt = ${spl_opt}\n"
printf "tpd_out = ${tpd_out}\n"
printf "var_xtr = ${var_xtr}\n"
printf "ypf_max = ${ypf_max}\n"
printf "yr_nbr = ${yr_nbr}\n"
printf "yr_sbs = ${yr_sbs}\n"
fi # !dbg
fi # !clm_md
if [ ${dbg_lvl} -ge 2 ]; then
printf "dbg: Will split into files for ${var_nbr} variable(s):\n"
for ((var_idx=0;var_idx<${var_nbr};var_idx++)); do
printf "${var_sbs[${var_idx}]}\n"
done # !var_idx
printf "dbg: Will split timeseries into ${sgm_nbr} segment(s):\n"
for ((sgm_idx=0;sgm_idx<${sgm_nbr};sgm_idx++)); do
printf "Segment ${sgm_idx} months: ${yyyy_srt_sgm[${sgm_idx}]}${mm_srt_sgm[${sgm_idx}]}--${yyyy_end_sgm[${sgm_idx}]}${mm_end_sgm[${sgm_idx}]}\n"
done # !sgm_idx
# printf "dbg: Will split ${fl_nbr} files into ${sgm_nbr} segment(s):\n"
# for ((sgm_idx=0;sgm_idx<${sgm_nbr};sgm_idx++)); do
# printf "${fl_sgm[${sgm_idx}]}\n"
# done # !sgm_idx
fi # !dbg
# Create template output filenames (to avoid Bash 2D string arrays)
for ((sgm_idx=0;sgm_idx<sgm_nbr;sgm_idx++)); do
if [ -z "${fml_nm_usr}" ]; then
fl_out_tpl[${sgm_idx}]="${drc_out}/var_nm_tpl_${yyyy_srt_sgm[${sgm_idx}]}${mm_srt_sgm[${sgm_idx}]}_${yyyy_end_sgm[${sgm_idx}]}${mm_end_sgm[${sgm_idx}]}.nc"
else # !fml_nm_usr
fl_out_tpl[${sgm_idx}]="${drc_out}/var_nm_tpl_${fml_nm}_${yyyy_srt_sgm[${sgm_idx}]}${mm_srt_sgm[${sgm_idx}]}_${yyyy_end_sgm[${sgm_idx}]}${mm_end_sgm[${sgm_idx}]}.nc"
fi # !fml_nm_usr
if [ -n "${rgr_opt}" ]; then
fl_rgr_tpl[${sgm_idx}]="${fl_out_tpl[${sgm_idx}]/${drc_out}/${drc_rgr}}"
if [ "${drc_out}" = "${drc_rgr}" ]; then
# Append geometry suffix to regridded files in same directory as native climo
# http://tldp.org/LDP/abs/html/string-manipulation.html
dfl_sfx='rgr'
rgr_sfx=`expr match "${rgr_map}" '.*_to_\(.*\).nc'`
if [ "${#rgr_sfx}" -eq 0 ]; then
printf "${spt_nm}: WARNING Unable to extract geometric suffix from mapfile, will suffix regridded files with \"${dfl_sfx}\" instead\n"
rgr_sfx=${dfl_sfx}
else
yyyymmdd_sng=`expr match "${rgr_sfx}" '.*\(\.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]\)'` # Find YYYYYMMDD
if [ "${#yyyymmdd_sng}" -ne 0 ]; then
rgr_sfx=${rgr_sfx%%${yyyymmdd_sng}} # Delete YYYYYMMDD
fi # !strlen
fi # !strlen
# rgr_sfx=`expr match "${rgr_sfx}" '\(.*\)\.[0-9][0-9][0-9][0-9][0-9][0-9]'` #
fl_rgr_tpl[${sgm_idx}]="${fl_rgr_tpl[${sgm_idx}]/.nc/_${rgr_sfx}.nc}"
fi # !drc_rgr
fi # !rgr_opt
done # !sgm_idx
# Begin outer loop over time segments
for ((sgm_idx=0;sgm_idx<sgm_nbr;sgm_idx++)); do
# Begin loop over variables to split
let bch_nbr=$((var_nbr / job_nbr))
let bch_flg=$((var_nbr % job_nbr))
if [ ${bch_nbr} -eq 0 ]; then
let bch_nbr=$((bch_nbr+1))
elif [ ${bch_flg} -ne 0 ]; then
let bch_nbr=$((bch_nbr+1))
fi # !bch_flg
let bch_nbrm1=$((bch_nbr-1))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# var_idx is 0-based, bch_idx is 0-based
let var_idx_srt=$((bch_idx * job_nbr))
let var_idx_end=$((var_idx_srt + job_nbr - 1))
if [ ${bch_idx} -eq ${bch_nbrm1} ] && [ ${bch_flg} -ne 0 ]; then
let var_idx_srt=$((bch_idx * job_nbr))
let var_idx_end=$((var_nbr - 1))
fi # !bch_flg
for ((var_idx=var_idx_srt;var_idx<=var_idx_end;var_idx++)); do
fl_out[${var_idx}]="${fl_out_tpl[${sgm_idx}]/var_nm_tpl/${var_sbs[${var_idx}]}}"
if [ "${clm_md}" = 'hfs' ]; then
# 20200619 High-frequency output requires time-coordinate hyperslabs
if [ "${mth_flg}" = 'Yes' ]; then
tm_srt="${yyyy_srt_sgm[${sgm_idx}]}-${mm_srt_sgm[${sgm_idx}]}-01T00:00:01"
tm_end="${yyyy_end_sgm[${sgm_idx}]}-${mm_endp1_sgm[${sgm_idx}]}-01T00:00:00"
else # !mth_flg
tm_srt="${yyyy_srt_sgm[${sgm_idx}]}-01-01T00:00:01"
tm_end="${yyyy_endp1_sgm[${sgm_idx}]}-01-01T00:00:00"
fi # !mth_flg
if [ ${tpd_out} -lt 3600 ]; then
cmd_sbs[${var_idx}]="${cmd_mpi[${var_idx}]} OMP_PROC_BIND=false ncrcat -O -v ${var_sbs[${var_idx}]}${var_xtr} ${nco_opt} ${spl_opt} ${gaa_sng} -d time,${tm_srt},${tm_end} ${ppn_opt} ${fl_sgm[${sgm_idx}]} ${fl_out[${var_idx}]}"
else
# Deprecated code to output in sub-average periods or different shapes
fl_rcat[${var_idx}]="${drc_out}/${var_sbs[${var_idx}]}_${yyyy_srt_sgm[${sgm_idx}]}01_${yyyy_end_sgm[${sgm_idx}]}12_ncrcat.nc"
fl_ecat[${var_idx}]="${drc_out}/${var_sbs[${var_idx}]}_${yyyy_srt_sgm[${sgm_idx}]}01_${yyyy_end_sgm[${sgm_idx}]}12_ncecat.nc"
fl_pdq[${var_idx}]="${drc_out}/${var_sbs[${var_idx}]}_${yyyy_srt_sgm[${sgm_idx}]}01_${yyyy_end_sgm[${sgm_idx}]}12_ncpdq.nc"
fl_tpd_tpl="${drc_out}/${var_sbs[${var_idx}]}_${yyyy_srt_sgm[${sgm_idx}]}01_${yyyy_end_sgm[${sgm_idx}]}12_tpd"
fl_tpd_all=''
for ((tpd_idx=0;tpd_idx<tpd_out;tpd_idx++)); do
fl_tpd[${tpd_idx}]="${drc_out}/${var_sbs[${var_idx}]}_${yyyy_srt_sgm[${sgm_idx}]}01_${yyyy_end_sgm[${sgm_idx}]}12_tpd${tpd_idx}.nc"
fl_tpd_all="${fl_tpd_all} ${fl_tpd[${tpd_idx}]}"
done # !tpd_idx
# 20200627: Average all timesteps into one climatological record as data(time,space)
# cmd_sbs[${var_idx}]="${cmd_mpi[${var_idx}]} OMP_PROC_BIND=false ncrcat -O -v ${var_sbs[${var_idx}]}${var_xtr} ${nco_opt} ${spl_opt} ${gaa_sng} -d time,${tm_srt},${tm_end} ${ppn_opt} ${fl_sgm[${sgm_idx}]} ${fl_rcat[${var_idx}]};/bin/rm -f ${fl_out[${var_idx}]};for srt in `seq 0 $((tpd_out-1))`; do ncra --rec_apn -d time,\${srt},,${tpd_out} ${nco_opt} ${fl_rcat[${var_idx}]} ${fl_out[${var_idx}]};done;/bin/rm -f ${fl_rcat[${var_idx}]}"
# 20200628: Output one record per year as data(time,tpd,space), easier to regrid
cmd_sbs[${var_idx}]="${cmd_mpi[${var_idx}]} OMP_PROC_BIND=false ncrcat -O -v ${var_sbs[${var_idx}]}${var_xtr} ${nco_opt} ${spl_opt} ${gaa_sng} -d time,${tm_srt},${tm_end} ${ppn_opt} ${fl_sgm[${sgm_idx}]} ${fl_rcat[${var_idx}]};/bin/rm -f ${fl_out[${var_idx}]};for srt in `seq 0 $((tpd_out-1))`; do ncrcat -O -v ${var_sbs[${var_idx}]}${var_xtr} -d time,\${srt},,${tpd_out} ${ppn_opt} ${fl_rcat[${var_idx}]} ${fl_tpd_tpl}\${srt}.nc;ncra -O --mro -d time,,,${dpy},${dpy} ${fl_tpd_tpl}\${srt}.nc ${fl_tpd_tpl}\${srt}.nc;done;/bin/rm -f ${fl_rcat[${var_idx}]};ncecat -O -u hour ${nco_opt} ${fl_tpd_all} ${fl_ecat[${var_idx}]};/bin/rm -f ${fl_tpd_all};ncap2 -O ${nco_opt} -s 'hour=array(0.0,24/\$hour.size,\$hour);*hr_dlt=(hour(1)-hour(0))/2;hour+=hr_dlt;hour@units=\"hours\";hour@long_name=\"Hour of Day\";hour_bnds=make_bounds(hour,\$nbnd,\"hour_bnds\")' ${fl_ecat[${var_idx}]} ${fl_pdq[${var_idx}]};/bin/rm -f ${fl_ecat[${var_idx}]};ncpdq -O ${nco_opt} -a time,hour ${fl_pdq[${var_idx}]} ${fl_out[${var_idx}]};/bin/rm -f ${fl_pdq[${var_idx}]}"
fi # !tpd_out
else
# Normal splitting, non-high-frequency
cmd_sbs[${var_idx}]="${cmd_mpi[${var_idx}]} OMP_PROC_BIND=false ncrcat -O -v ${var_sbs[${var_idx}]}${var_xtr} ${nco_opt} ${spl_opt} ${gaa_sng} ${ppn_opt} ${fl_sgm[${sgm_idx}]} ${fl_out[${var_idx}]}"
fi # !clm_md
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_sbs[${var_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_sbs[${var_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR subset cmd_sbs[${var_idx}] failed. Debug this:\n${cmd_sbs[${var_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_sbs[${var_idx}]} ${par_opt}
sbs_pid[${var_idx}]=$!
fi # !par_opt
fi # !dbg
done # !var_idx
# Block NNN: Wait
# Parallel splitting (both Background and MPI) spawns simultaneous processes in batches of ${job_nbr}
# Once ${job_nbr} jobs are running, wait() for all to finish before issuing another batch
if [ -n "${par_opt}" ]; then
#printf "${spt_nm}: DEBUG var_idx = ${var_idx}, bch_idx = ${bch_idx}, bch_flg = ${bch_flg}\n"
for ((var_idx=var_idx_srt;var_idx<=var_idx_end;var_idx++)); do
wait ${sbs_pid[${var_idx}]}
if [ "$?" -eq 0 ]; then
printf "`date`: Generated ${cmd_sbs[${var_idx}]##* }\n"
else
printf "${spt_nm}: ERROR Failed to split. cmd_sbs[${var_idx}] failed. Debug this:\n${cmd_sbs[${var_idx}]}\n"
# 20190323: exiting parent here creates orphans, kill live children then exit
for ((kid_idx=var_idx+1;kid_idx<=var_idx_end;kid_idx++)); do
kill -9 ${sbs_pid[${kid_idx}]}
done # !kid_idx
exit 1
fi # !err
done # !var_idx
fi # !par_opt
if [ "${rgn_stt}" != 'No' ]; then
for ((var_idx=var_idx_srt;var_idx<=var_idx_end;var_idx++)); do
# 20250610: Until today we always computed regional statistics of all variables in splitter list
# However, regional statistics are fragile and break on non-horozontal variables
# Moreover, it is difficult for users to know which of the potentially hundreds of variables are not horizontal
# Starting today we always check first and skip regional statistics for non-horizontal variables
is_hrz[${var_idx}]=`ncks --is_hrz ${var_sbs[${var_idx}]} ${fl_out[${var_idx}]}`
if [ ${is_hrz[${var_idx}]} = 'No' ]; then
echo "${spt_nm}: WARNING Splitter is skipping regional statistics for variable ${var_sbs[${var_idx}]} which is not a purely horizontal timeseries"
continue;
fi # !is_hrz
cmd_rgn="*rgn_nbr=3;defdim(\"rgn\",rgn_nbr);*${var_sbs[${var_idx}]}_tmp=0.0f*${var_sbs[${var_idx}]}.avg(${hrz_arg});*${var_sbs[${var_idx}]}_rgn[time,rgn]=${var_sbs[${var_idx}]}_tmp;${var_sbs[${var_idx}]}_rgn@coordinates=\"region_name\";*${lat_nm}_area=${lat_nm}+0.0*${area_nm};*msk_sth=0*${lat_nm}_area.int();delete_miss(msk_sth);*msk_nrt=0*${lat_nm}_area.int();delete_miss(msk_nrt);*idx_glb=0;*idx_nrt=1;*idx_sth=2;*rgn_len=19;defdim(\"rgn_len\",rgn_len);region_name[rgn,rgn_len]=\" \";region_name(idx_glb,0:5)=\"Global\";region_name(idx_nrt,:)=\"Northern Hemisphere\";region_name(idx_sth,:)=\"Southern Hemisphere\";if(${flg_sum}) region_name@long_name=\"${var_sbs[${var_idx}]} timeseries array contains area-weighted sums over these regions\"; else region_name@long_name=\"${var_sbs[${var_idx}]} timeseries array contains area-weighted averages over these regions\";where(${lat_nm}_area < 0.0) msk_sth=1; elsewhere msk_nrt=1;${var_sbs[${var_idx}]}_rgn(:,idx_glb)=((${var_sbs[${var_idx}]}*${area_wgt}).avg(${hrz_arg})/(${area_wgt}).avg(${hrz_arg})).float();${var_sbs[${var_idx}]}_rgn(:,idx_nrt)=((${var_sbs[${var_idx}]}*${area_wgt}*msk_nrt).avg(${hrz_arg})/(${area_wgt}*msk_nrt).avg(${hrz_arg})).float();${var_sbs[${var_idx}]}_rgn(:,idx_sth)=((${var_sbs[${var_idx}]}*${area_wgt}*msk_sth).avg(${hrz_arg})/(${area_wgt}*msk_sth).avg(${hrz_arg})).float();if(${flg_sum}){${var_sbs[${var_idx}]}_rgn(:,idx_glb)=${var_sbs[${var_idx}]}_rgn(:,idx_glb)*(${area_wgt}).total(${hrz_arg}).float()*${sum_scl};${var_sbs[${var_idx}]}_rgn(:,idx_nrt)=${var_sbs[${var_idx}]}_rgn(:,idx_nrt)*(${area_wgt}*msk_nrt).total(${hrz_arg}).float()*${sum_scl};${var_sbs[${var_idx}]}_rgn(:,idx_sth)=${var_sbs[${var_idx}]}_rgn(:,idx_sth)*(${area_wgt}*msk_sth).total(${hrz_arg}).float()*${sum_scl};}${var_sbs[${var_idx}]}=${var_sbs[${var_idx}]}_rgn;push(&${var_sbs[${var_idx}]}@cell_methods,\" area: mean\");if(exists(time_bnds)) time_bnds=time_bnds;if(exists(time_bounds)) time_bounds=time_bounds;valid_area_per_gridcell=${area_wgt};"
# NB: We do not pass nco_opt to ncap2 here
# nco_opt might contain switches like -x that have different meanings in ncap2
# File format is inherited from timeseries file, so no need to pass that
# One option to consider passing by default here is --no_tmp_fl
# 20240411: Do not append history for rgn_stt computation---too much like encoding a program in a dataset
# 20241218: Behavior before today was to pass -v to ncap2 in rgn_stt computation
# However, this prevented output of --var_xtr fields. We omit '-v' as of NCO 5.3.0.
# Pros: More flexible, allows users to request e.g., --var_xtr=area,landfrac for scaling
# Cons: Larger files, and users might pollute --var_xtr with timeseries other timeseries
cmd_stt[${var_idx}]="${cmd_mpi[${var_idx}]} OMP_PROC_BIND=false ncap2 -h -O -s '${cmd_rgn}' ${fl_out[${var_idx}]} ${fl_out[${var_idx}]}"
# 20220817: Prior to today, --glb_avg used ncwa for global average
# As of NCO 5.1.1, --glb_avg is a deprecated synonym for --rgn_avg
# As of NCO 5.2.8, --rgn_avg is a synonym for --rgn_stt=avg
# Both produce (currently three) regional timeseries
# The original --glb_avg implementation was:
# cmd_stt[${var_idx}]="${cmd_mpi[${var_idx}]} OMP_PROC_BIND=false ncwa -O -w ${area_nm} -a ${hrz_dmn} ${fl_out[${var_idx}]} ${fl_out[${var_idx}]}"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_stt[${var_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_stt[${var_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR global and regional statistics cmd_stt[${var_idx}] failed. Debug this:\n${cmd_stt[${var_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_stt[${var_idx}]} ${par_opt} # eval always returns 0 on backgrounded processes
avg_pid[${var_idx}]=$!
fi # !par_opt
fi # !dbg
done # !var_idx
# Once ${job_nbr} jobs are running, wait() for all to finish before issuing another batch
if [ -n "${par_opt}" ]; then
for ((var_idx=var_idx_srt;var_idx<=var_idx_end;var_idx++)); do
if [ ${is_hrz[${var_idx}]} = 'Yes' ]; then
wait ${avg_pid[${var_idx}]}
if [ "$?" -eq 0 ]; then
printf "`date`: Global and regional statistics ${cmd_stt[${var_idx}]##* }\n"
else
printf "${spt_nm}: ERROR Failed in global and regional statistics. cmd_stt[${var_idx}] failed. Debug this:\n${cmd_stt[${var_idx}]}\n"
# 20190323: exiting parent here creates orphans, kill live children then exit
for ((kid_idx=var_idx+1;kid_idx<=var_idx_end;kid_idx++)); do
kill -9 ${avg_pid[${kid_idx}]}
done # !kid_idx
exit 1
fi # !err
fi # !is_hrz
done # !var_idx
fi # !par_opt
fi # !rgn_stt
done # !bch_idx
# Begin loop over variables to regrid
if [ -n "${rgr_opt}" ]; then
let bch_nbr=$((var_nbr / job_nbr))
let bch_flg=$((var_nbr % job_nbr))
if [ ${bch_nbr} -eq 0 ]; then
let bch_nbr=$((bch_nbr+1))
elif [ ${bch_flg} -ne 0 ]; then
let bch_nbr=$((bch_nbr+1))
fi # !bch_flg
let bch_nbrm1=$((bch_nbr-1))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# var_idx is 0-based, bch_idx is 0-based
let var_idx_srt=$((bch_idx * job_nbr))
let var_idx_end=$((var_idx_srt + job_nbr - 1))
if [ ${bch_idx} -eq ${bch_nbrm1} ] && [ ${bch_flg} -ne 0 ]; then
let var_idx_srt=$((bch_idx * job_nbr))
let var_idx_end=$((var_nbr - 1))
fi # !bch_flg
for ((var_idx=var_idx_srt;var_idx<=var_idx_end;var_idx++)); do
fl_rgr[${var_idx}]="${fl_rgr_tpl[${sgm_idx}]/var_nm_tpl/${var_sbs[${var_idx}]}}"
#cmd_rgr[${var_idx}]="${cmd_mpi[${var_idx}]} ncks -O -t ${thr_nbr} ${nco_opt} ${rgr_opt} ${spl_opt} ${spl_rgr_opt} ${fl_out[${var_idx}]} ${fl_rgr[${var_idx}]}"
cmd_rgr[${var_idx}]="${cmd_mpi[${var_idx}]} ncremap -u .pid${spt_pid}.split.${var_idx}.tmp --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${spl_opt} ${spl_rgr_opt} ${prc_opt} ${rmp_opt} ${fl_out[${var_idx}]} ${fl_rgr[${var_idx}]}"
if [ "${mdl_typ}" = 'mpas' ]; then
# 20181114: Remove '-C' argument from ncremap invocation for MPAS splitter only
# This ensures CMIP6 MPAS timeseries will have _FillValue
cmd_rgr[${var_idx}]="${cmd_mpi[${var_idx}]} ncremap -u .pid${spt_pid}.split.${var_idx}.tmp ${prc_opt} ${d2f_opt} ${dpt_opt} --nco_opt=\"${nco_opt}\" ${spl_opt} ${spl_rgr_opt} -t ${thr_nbr} ${map_opt} ${vrt_opt} ${fl_out[${var_idx}]} ${fl_rgr[${var_idx}]}"
fi # !mdl_typ
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_rgr[${var_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_rgr[${var_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR regrid cmd_rgr[${var_idx}] failed. Debug this:\n${cmd_rgr[${var_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_rgr[${var_idx}]} ${par_opt}
rgr_pid[${var_idx}]=$!
fi # !par_opt
fi # !dbg
done # !var_idx
# Block NNN: Wait
# Parallel regridding (both Background and MPI) spawns simultaneous processes in batches of ${job_nbr}
# Once ${job_nbr} jobs are running, wait() for all to finish before issuing another batch
if [ -n "${par_opt}" ]; then
for ((var_idx=var_idx_srt;var_idx<=var_idx_end;var_idx++)); do
wait ${rgr_pid[${var_idx}]}
if [ "$?" -eq 0 ]; then
printf "`date`: Regridded ${cmd_rgr[${var_idx}]##* }\n"
else
printf "${spt_nm}: ERROR Failed to regrid. cmd_rgr[${var_idx}] failed. Debug this:\n${cmd_rgr[${var_idx}]}\n"
# 20190323: exiting parent here creates orphans, kill live children then exit
for ((kid_idx=var_idx+1;kid_idx<=var_idx_end;kid_idx++)); do
kill -9 ${rgr_pid[${kid_idx}]}
done # !kid_idx
exit 1
fi # !err
done # !var_idx
fi # !par_opt
done # !bch_idx
if [ "${no_ntv_tms}" = 'Yes' ]; then
# Omit native-grid split timeseries by overwriting fl_out with fl_rgr
for ((var_idx=0;var_idx<var_nbr;var_idx++)); do
/bin/mv -f ${fl_rgr[${var_idx}]} ${fl_out[${var_idx}]}
printf "`date`: Overwrote native timeseries with regridded for ${var_sbs[${var_idx}]}\n"
done # !var_idx
fi # !no_ntv_tms
fi # !rgr_opt
done # !sgm_idx
fi # !tms_flg
# Block 1 Loop 2: Climatologies based on annual means
if [ "${clm_flg}" = 'Yes' ] && [ "${clm_md}" = 'ann' ]; then
clm_idx=1
fl_all=''
for ((fl_idx=0;fl_idx<fl_nbr;fl_idx++)); do
fl_all="${fl_all} ${fl_in[${fl_idx}]}"
done # !fl
# Check for existence of raw input only when file will be used (NB: next ~12 lines duplicate monthly code)
if [ "${bnr_flg}" = 'No' ]; then
for fl_crr in ${fl_all} ; do
if [ ! -f "${drc_in}/${fl_crr}" ] && [ ! -L "${drc_in}/${fl_crr}" ]; then
echo "${spt_nm}: ERROR Unable to find required input file ${drc_in}/${fl_crr}"
echo "${spt_nm}: HINT All files implied to exist by the climatology bounds (start/end year/month) and by the specified (with -P or -m) or default model type, must be in ${drc_in} before ${spt_nm} will proceed"
exit 1
fi # ! -f
done # !fl_crr
else # !bnr_flg
# In binary mode drc_out is actually used to locate input files from climatology B (same as output files in incremental mode)
drc_out="${drc_in}"
fi # !bnr_flg
fl_out[${clm_idx}]="${drc_out}/${out_nm}_ANN_${yyyy_srt}01_${yyyy_end}12_climo.nc"
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra -O ${nco_opt} ${gaa_sng} -p ${drc_in} ${fl_all} ${fl_out[${clm_idx}]} ${par_opt}"
# Block 1 Loop 2: Climatological annual mean
printf "Climatological annual mean...\n"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_clm[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
eval ${cmd_clm[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR annual climo cmd_clm[${clm_idx}] failed\n"
exit 1
fi # !err
fi # !dbg
wait
# Block 2: Regrid climatological annual mean
if [ -n "${rgr_opt}" ]; then
printf "Regrid annual data...\n"
cmd_rgr[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncks -O -t ${thr_nbr} ${nco_opt} ${rgr_opt} ${fl_out[${clm_idx}]} ${fl_out[${clm_idx}]/.nc/.rgr.nc}"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_rgr[${clm_idx}]} ${par_opt}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
eval ${cmd_rgr[${clm_idx}]} ${par_opt}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR annual regrid cmd_rgr[${clm_idx}] failed\n"
exit 1
fi # !err
fi # !dbg
wait
printf "Done with regridding\n"
fi # !rgr_opt
fi # !ann
# Block 1 Loop 2: Climatologies based on daily means
if [ "${clm_flg}" = 'Yes' ] && [ "${clm_md}" = 'dly' ]; then
fl_all=''
for ((fl_idx=0;fl_idx<fl_nbr;fl_idx++)); do
fl_all="${fl_all} ${fl_in[${fl_idx}]}"
done # !fl
let srd=${dpy}*${tpd_out}
drn=${tpd_out}
yyyy_srt=`printf "%04d" ${yr_srt}`
yyyy_end=`printf "%04d" ${yr_end}`
# 20200625 Off-by-one bug?
# let yr_endp1=${yr_end}+1
# yyyy_endp1=`printf "%04d" ${yr_endp1}`
clm_idx=0
for mth in `seq 1 12`; do
MM=`printf "%02d" ${mth}`
for day in `seq 1 ${dpm[${mth}]}`; do
DD=`printf "%02d" ${day}`
fl_out[${clm_idx}]="${drc_out}/${out_nm}_${yyyy_srt}${MM}${DD}_${yyyy_end}${MM}${DD}_climo.nc"
fl_cat[${clm_idx}]="${drc_out}/${out_nm}_${yyyy_srt}${MM}${DD}_${yyyy_end}${MM}${DD}_ncrcat.nc"
fl_rgr[${clm_idx}]="${fl_out[${clm_idx}]/${drc_out}/${drc_rgr}}"
tm_srt="${yyyy_srt}-${MM}-${DD}T00:00:00"
tm_end="${yyyy_end}-${MM}-${DD}T23:59:59"
if [ ${tpd_out} -eq 1 ]; then
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncra -O ${nco_opt} ${gaa_sng} -d time,${tm_srt},${tm_end},${srd},1 ${ppn_opt} ${fl_all} ${fl_out[${clm_idx}]}"
else
# 20170619: Diurnal mode is really slow on MacOS (UDUnits parsing sluggishness?)
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} OMP_PROC_BIND=false ncrcat -O ${nco_opt} ${gaa_sng} -d time,${tm_srt},${tm_end},${srd},${drn} ${ppn_opt} ${fl_all} ${fl_cat[${clm_idx}]};/bin/rm -f ${fl_out[${clm_idx}]};for srt in `seq 0 $((tpd_out-1))`; do ncra --rec_apn -d time,\${srt},,${tpd_out} ${nco_opt} ${gaa_sng} ${fl_cat[${clm_idx}]} ${fl_out[${clm_idx}]};done;/bin/rm -f ${fl_cat[${clm_idx}]}"
fi # !tpd
let clm_idx=${clm_idx}+1
done # !day
done # !mth
if [ -n "${rgr_opt}" ]; then
for ((clm_idx=0;clm_idx<clm_nbr;clm_idx++)); do
fl_rgr[${clm_idx}]="${fl_out[${clm_idx}]/${drc_out}/${drc_rgr}}"
if [ "${drc_out}" = "${drc_rgr}" ]; then
# Append geometry suffix to regridded files in same directory as native climo
# http://tldp.org/LDP/abs/html/string-manipulation.html
dfl_sfx='rgr'
rgr_sfx=`expr match "${rgr_map}" '.*_to_\(.*\).nc'`
if [ "${#rgr_sfx}" -eq 0 ]; then
printf "${spt_nm}: WARNING Unable to extract geometric suffix from mapfile, will suffix regridded files with \"${dfl_sfx}\" instead\n"
rgr_sfx=${dfl_sfx}
else
yyyymmdd_sng=`expr match "${rgr_sfx}" '.*\(\.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]\)'` # Find YYYYYMMDD
if [ "${#yyyymmdd_sng}" -ne 0 ]; then
rgr_sfx=${rgr_sfx%%${yyyymmdd_sng}} # Delete YYYYYMMDD
fi # !strlen
fi # !strlen
# rgr_sfx=`expr match "${rgr_sfx}" '\(.*\)\.[0-9][0-9][0-9][0-9][0-9][0-9]'` #
fl_rgr[${clm_idx}]="${fl_rgr[${clm_idx}]/.nc/_${rgr_sfx}.nc}"
fi # !drc_rgr
done # !clm_idx
fi # !rgr_opt
# Begin loop over days to climatologize
printf "Climatological daily mean...\n"
let bch_nbr=$((clm_nbr / job_nbr))
let bch_flg=$((clm_nbr % job_nbr))
if [ ${bch_nbr} -eq 0 ]; then
let bch_nbr=$((bch_nbr+1))
elif [ ${bch_flg} -ne 0 ]; then
let bch_nbr=$((bch_nbr+1))
fi # !bch_flg
let bch_nbrm1=$((bch_nbr-1))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# clm_idx is 0-based, bch_idx is 0-based
let clm_idx_srt=$((bch_idx * job_nbr))
let clm_idx_end=$((clm_idx_srt + job_nbr - 1))
if [ ${bch_idx} -eq ${bch_nbrm1} ] && [ ${bch_flg} -ne 0 ]; then
let clm_idx_srt=$((bch_idx * job_nbr))
let clm_idx_end=$((clm_nbr - 1))
fi # !bch_flg
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_clm[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_clm[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR daily cmd_clm[${clm_idx}] failed. Debug this:\n${cmd_clm[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_clm[${clm_idx}]} ${par_opt}
clm_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !fl_idx
# Block NNN: Wait
# Parallel splitting (both Background and MPI) spawns simultaneous processes in batches of ${job_nbr}
# Once ${job_nbr} jobs are running, wait() for all to finish before issuing another batch
if [ -n "${par_opt}" ]; then
if [ ${dbg_lvl} -ge 1 ] && [ ${clm_idx_srt} -le ${clm_idx_end} ]; then
printf "${spt_nm}: Waiting for batch ${bch_idx} to finish at clm_idx = ${clm_idx}...\n"
fi # !dbg
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
wait ${clm_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR Failed daily average. cmd_clm[${clm_idx}] failed. Debug this:\n${cmd_clm[${clm_idx}]}\n"
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
done # !bch_idx
# Begin loop over days to regrid
if [ -n "${rgr_opt}" ]; then
let bch_nbr=$((clm_nbr / job_nbr))
let bch_flg=$((clm_nbr % job_nbr))
if [ ${bch_nbr} -eq 0 ]; then
let bch_nbr=$((bch_nbr+1))
elif [ ${bch_flg} -ne 0 ]; then
let bch_nbr=$((bch_nbr+1))
fi # !bch_flg
let bch_nbrm1=$((bch_nbr-1))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# clm_idx is 0-based, bch_idx is 0-based
let clm_idx_srt=$((bch_idx * job_nbr))
let clm_idx_end=$((clm_idx_srt + job_nbr - 1))
if [ ${bch_idx} -eq ${bch_nbrm1} ] && [ ${bch_flg} -ne 0 ]; then
let clm_idx_srt=$((bch_idx * job_nbr))
let clm_idx_end=$((clm_nbr - 1))
fi # !bch_flg
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
cmd_rgr[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncks -O -t ${thr_nbr} ${nco_opt} ${rgr_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
if [ "${mdl_typ}" = 'mpas' ]; then
cmd_rgr[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncremap -C -u .pid${spt_pid}.daily.${clm_idx}.tmp ${prc_opt} ${d2f_opt} ${dpt_opt} --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${map_opt} ${vrt_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
fi # !mdl_typ
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_rgr[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_rgr[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR regrid cmd_rgr[${clm_idx}] failed. Debug this:\n${cmd_rgr[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_rgr[${clm_idx}]} ${par_opt}
rgr_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
# Block NNN: Wait
# Parallel regridding (both Background and MPI) spawns simultaneous processes in batches of ${job_nbr}
# Once ${job_nbr} jobs are running, wait() for all to finish before issuing another batch
if [ -n "${par_opt}" ]; then
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
wait ${rgr_pid[${clm_idx}]}
if [ "$?" -eq 0 ]; then
printf "`date`: Regridded ${cmd_rgr[${clm_idx}]##* }\n"
else
printf "${spt_nm}: ERROR Failed to regrid. cmd_rgr[${clm_idx}] failed. Debug this:\n${cmd_rgr[${clm_idx}]}\n"
# 20190323: exiting parent here creates orphans, kill live children then exit
for ((kid_idx=clm_idx+1;kid_idx<=clm_idx_end;kid_idx++)); do
kill -9 ${rgr_pid[${kid_idx}]}
done # !kid_idx
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
done # !bch_idx
fi # !rgr_opt
fi # !dly
# Many subsequent blocks only execute for normal and incremental monthly climos, not for binary climos, or non-monthly climos
if [ "${clm_flg}" = 'Yes' ] && [ "${clm_hfc_or_mth}" = 'Yes' ] && [ "${bnr_flg}" = 'No' ]; then
# Block 1 Loop 2: Execute and/or echo monthly climatology commands
printf "Generating climatology...\n"
[[ ${dbg_lvl} -ge 1 ]] && date_clm=$(date +"%s")
if [ ${job_nbr} -eq 1 ] || [ ${job_nbr} -eq 2 ] || [ ${job_nbr} -eq 3 ] || [ ${job_nbr} -eq 4 ] || [ ${job_nbr} -eq 6 ] || [ ${job_nbr} -eq 12 ]; then
echo "Successful match of job_nbr to list of permissible values" > /dev/null
else
echo "${spt_nm}: ERROR Job number job_nbr=${job_nbr} is invalid in monthly climo mode. Must be a factor of 12. Valid values are 1, 2, 3, 4, 6, and 12. Please re-submit with valid job_nbr."
exit 1
fi # !job_nbr
# Batch number is inverse to job number, so job_nbr=1->bch_nbr=12 (serial mode, slowest), job_nbr=2->bch_nbr=6, ... job_nbr=12->bch_nbr=1 (full background mode, fastest)
let bch_nbr=$((12 / job_nbr))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# clm_idx is 1-based, bch_idx is 0-based
let clm_idx_srt=$(((bch_idx * job_nbr) + 1))
let clm_idx_end=$((clm_idx_srt + job_nbr - 1))
#printf "${spt_nm}: DEBUG job_nbr = ${job_nbr}, bch_nbr = ${bch_nbr}, bch_idx = ${bch_idx}, clm_idx_srt = ${clm_idx_end}, clm_idx_end = ${clm_idx_end}\n"
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
printf "Climatological monthly mean for month ${clm_idx} ...\n"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_clm[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_clm[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly climo cmd_clm[${clm_idx}] failed. Debug this:\n${cmd_clm[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_clm[${clm_idx}]} ${par_opt} # eval always returns 0 on backgrounded processes
clm_pid[${clm_idx}]=$!
# Potential alternatives to eval:
# eval "${cmd_clm[${clm_idx}]}" # borken
# ${cmd_clm[${clm_idx}]} # borken
# "${cmd_clm[${clm_idx}]}" # borken
# exec "${cmd_clm[${clm_idx}]}" # borken
# $(${cmd_clm[${clm_idx}]}) # borken
# $("${cmd_clm[${clm_idx}]}") # works (when & inside cmd quotes)
fi # !par_opt
fi # !dbg
done # !clm_idx
if [ -n "${par_opt}" ]; then
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
wait ${clm_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly climo cmd_clm[${clm_idx}] failed. Debug this:\n${cmd_clm[${clm_idx}]}\n"
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
done # !bch_idx
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_clm))
echo "Elapsed time to generate monthly climos $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
# Clean-up monthly inputs (important since these file can be huge)
if [ ${clm_md} = 'hfc' ]; then
if [ "${fl_nbr}" -eq 1 ]; then
/bin/rm -f ${drc_out}/${out_nm}_[01][0-9]_tms_foo_${yyyy_clm_srt}_${yyyy_clm_end}.nc
else
/bin/rm -f ${drc_out}/${out_nm}_[01][0-9]_tpd[0-9][0-9]_*.nc
fi # !fl_nbr
fi # !clm_md
# Block 1: Loop 4: Regrid first twelve files. Load-balance by using idle nodes (nodes not used for seasonal climatologies).
if [ -n "${rgr_opt}" ]; then
printf "Regrid monthly data...\n"
[[ ${dbg_lvl} -ge 1 ]] && date_rgr=$(date +"%s")
for ((clm_idx=1;clm_idx<=12;clm_idx++)); do
# NB: Months, seasons, files are 1-based ([1..12], [13..16], [1..17]), nodes are 0-based ([0..11])
let nd_idx=$(((clm_idx-1+csn_nbr) % nd_nbr))
if [ ${nd_idx} -lt ${csn_nbr} ]; then
let nd_idx=${nd_idx}+${csn_nbr}
# 20190315: Not necessary to modulo final result by nd_nbr to handle csn_nbr > nd_nbr
# because (1-based) cmd_mpi array is always at least 13 elements long and each element is
# already round-robin'ed onto existing nodes
fi # !nd
# printf "dbg: clm_idx = ${clm_idx}, nd_idx = ${nd_idx}, csn_nbr = ${csn_nbr}, nd_nbr = ${nd_nbr}\n"
#cmd_rgr[${clm_idx}]="${cmd_mpi[${nd_idx}]} ncks -O -t ${thr_nbr} ${nco_opt} ${rgr_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
cmd_rgr[${clm_idx}]="${cmd_mpi[${nd_idx}]} ncremap -u .pid${spt_pid}.climo.${clm_idx}.tmp --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${prc_opt} ${rmp_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
if [ "${mdl_typ}" = 'mpas' ]; then
cmd_rgr[${clm_idx}]="${cmd_mpi[${nd_idx}]} ncremap -C -u .pid${spt_pid}.climo.${clm_idx}.tmp ${prc_opt} ${d2f_opt} ${dpt_opt} --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${map_opt} ${vrt_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
fi # !mdl_typ
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_rgr[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_rgr[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly regrid cmd_rgr[${clm_idx}] failed. Debug this:\n${cmd_rgr[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_rgr[${clm_idx}]} ${par_opt}
rgr_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
# Start seasonal means first, then wait() for monthly regridding to finish
fi # !rgr_opt
# Block 2: Climatological seasonal means
if [ "${csn_flg}" = 'Yes' ]; then
# Block 2 Loop 1: Generate seasonal commands
[[ ${dbg_lvl} -ge 1 ]] && date_clm=$(date +"%s")
# if [ ${clm_md} != 'hfc' ]; then
# for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
# clm_bnd[${clm_idx}]=''
# done # !clm_idx
# fi # !clm_md
for ((csn_idx=0;csn_idx<${csn_nbr};csn_idx++)); do
let clm_idx=1+12+${csn_idx}
fl_lst=''
for ((mth_idx=${csn_srt_idx[${map_r2d[${csn_idx}]}]};mth_idx<=${csn_end_idx[${map_r2d[${csn_idx}]}]};mth_idx++)); do
fl_lst="${fl_lst} ${fl_out[${mth_idx}]}"
done # !mth_idx
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ${cmd_mth2csn} ${clm_bnd[${clm_idx}]} -O -w ${csn_wgt[${map_r2d[${csn_idx}]}]} ${nco_opt} ${gaa_sng} ${fl_lst} ${fl_out[${clm_idx}]}"
# DJF specified as exception since month-order is non-consecutive (12,1,2)
if [ "${csn_abb[${map_r2d[${csn_idx}]}]}" = 'DJF' ]; then
cmd_clm[${clm_idx}]="${cmd_mpi[${clm_idx}]} ${cmd_mth2csn} ${clm_bnd[${clm_idx}]} -O -w 31,31,28 ${nco_opt} ${gaa_sng} ${fl_out[12]} ${fl_out[1]} ${fl_out[2]} ${fl_out[${clm_idx}]}"
fi # !DJF
done # !csn_idx
# cmd_clm[13]="${cmd_mpi[13]} ncra --cb -O -w 31,30,31 ${nco_opt} ${gaa_sng} ${fl_out[3]} ${fl_out[4]} ${fl_out[5]} ${fl_out[13]}"
# cmd_clm[14]="${cmd_mpi[14]} ncra --cb -O -w 30,31,31 ${nco_opt} ${gaa_sng} ${fl_out[6]} ${fl_out[7]} ${fl_out[8]} ${fl_out[14]}"
# cmd_clm[15]="${cmd_mpi[15]} ncra --cb -O -w 30,31,30 ${nco_opt} ${gaa_sng} ${fl_out[9]} ${fl_out[10]} ${fl_out[11]} ${fl_out[15]}"
# cmd_clm[16]="${cmd_mpi[16]} ncra --cb -O -w 31,31,28 ${nco_opt} ${gaa_sng} ${fl_out[12]} ${fl_out[1]} ${fl_out[2]} ${fl_out[16]}"
# Block 2 Loop 2: Execute and/or echo seasonal climatology commands
let bch_nbr=$((csn_nbr / job_nbr))
let bch_flg=$((csn_nbr % job_nbr))
if [ ${bch_nbr} -eq 0 ]; then
let bch_nbr=$((bch_nbr+1))
elif [ ${bch_flg} -ne 0 ]; then
let bch_nbr=$((bch_nbr+1))
fi # !bch_flg
let bch_nbrm1=$((bch_nbr-1))
for ((bch_idx=0;bch_idx<bch_nbr;bch_idx++)); do
# clm_idx is 1-based, bch_idx is 0-based
let clm_idx_srt=$((12 + (bch_idx * job_nbr) + 1))
let clm_idx_end=$((clm_idx_srt + job_nbr - 1))
if [ ${bch_idx} -eq ${bch_nbrm1} ] && [ ${bch_flg} -ne 0 ]; then
let clm_idx_srt=$((12 + (bch_idx * job_nbr) + 1))
let clm_idx_end=${clm_csn_end_idx}
fi # !bch_flg
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
let csn_idx=$((clm_idx - 12 - 1))
printf "Climatological seasonal mean for ${csn_abb[${map_r2d[${csn_idx}]}]} ...\n"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_clm[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_clm[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR seasonal climo cmd_clm[${clm_idx}] failed. Debug this:\n${cmd_clm[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_clm[${clm_idx}]} ${par_opt}
clm_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
if [ -n "${par_opt}" ]; then
for ((clm_idx=clm_idx_srt;clm_idx<=clm_idx_end;clm_idx++)); do
wait ${clm_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR seasonal climo cmd_clm[${clm_idx}] failed. Debug this:\n${cmd_clm[${clm_idx}]}\n"
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
done # !bch_idx
# wait() for seasonal climatologies to finish
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_clm))
echo "Elapsed time to generate seasonal climos $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
fi # !csn_flg
# wait() for monthly regridding, if any, to finish
if [ -n "${rgr_opt}" ]; then
if [ -n "${par_opt}" ]; then
for ((clm_idx=1;clm_idx<=12;clm_idx++)); do
wait ${rgr_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR monthly regrid cmd_rgr[${clm_idx}] failed. Debug this:\n${cmd_rgr[${clm_idx}]}\n"
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_rgr))
echo "Elapsed time to regrid monthly climos $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
fi # !rgr_opt
# Block 2: Loop 4: Regrid seasonal files. Load-balance by using idle nodes (nodes not used for annual mean).
if [ "${csn_flg}" = 'Yes' ]; then
if [ -n "${rgr_opt}" ]; then
printf "Regrid seasonal data...\n"
[[ ${dbg_lvl} -ge 1 ]] && date_rgr=$(date +"%s")
for ((clm_idx=${clm_csn_srt_idx};clm_idx<=${clm_csn_end_idx};clm_idx++)); do
let nd_idx=$(((clm_idx-1+csn_nbr) % nd_nbr))
if [ ${nd_idx} -lt ${csn_nbr} ]; then
let nd_idx=${nd_idx}+${csn_nbr}
fi # !nd
#cmd_rgr[${clm_idx}]="${cmd_mpi[${nd_idx}]} ncks -O -t ${thr_nbr} ${nco_opt} ${rgr_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
cmd_rgr[${clm_idx}]="${cmd_mpi[${nd_idx}]} ncremap -u .pid${spt_pid}.climo.${clm_idx}.tmp --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${prc_opt} ${rmp_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
if [ "${mdl_typ}" = 'mpas' ]; then
cmd_rgr[${clm_idx}]="${cmd_mpi[${nd_idx}]} ncremap -C -u .pid${spt_pid}.climo.${clm_idx}.tmp ${prc_opt} ${d2f_opt} ${dpt_opt} --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${map_opt} ${vrt_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
fi # !mdl_typ
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_rgr[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_rgr[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR seasonal regrid cmd_rgr[${clm_idx}] failed. Debug this:\n${cmd_rgr[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_rgr[${clm_idx}]} ${par_opt}
rgr_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
# Start annual mean first, then wait() for seasonal regridding to finish
fi # !rgr_opt
fi # !csn_flg
# Block 3: Climatological annual mean
if [ "${ann_flg}" = 'Yes' ]; then
printf "Climatological annual mean...\n"
[[ ${dbg_lvl} -ge 1 ]] && date_clm=$(date +"%s")
fl_lst="${fl_out[${clm_csn_mam_idx}]} ${fl_out[${clm_csn_jja_idx}]} ${fl_out[${clm_csn_son_idx}]} ${fl_out[${clm_csn_djf_idx}]}"
cmd_clm[${clm_nbr}]="${cmd_mpi[${clm_nbr}]} ${cmd_csn2ann} ${clm_bnd[${clm_nbr}]} -O -w 92,92,91,90 ${nco_opt} ${gaa_sng} ${fl_lst} ${fl_out[${clm_nbr}]}"
# cmd_clm[17]="${cmd_mpi[17]} ncra --c2b -O -w 92,92,91,90 ${nco_opt} ${gaa_sng} ${fl_out[13]} ${fl_out[14]} ${fl_out[15]} ${fl_out[16]} ${fl_out[17]}"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_clm[${clm_nbr}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_clm[${clm_nbr}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR annual climo cmd_clm[${clm_nbr}] failed. Debug this:\n${cmd_clm[${clm_nbr}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_clm[${clm_nbr}]} ${par_opt}
clm_pid[${clm_nbr}]=$!
fi # !par_opt
fi # !dbg
fi # !ann_flg
# wait() for seasonal regridding, if any, to finish
if [ "${csn_flg}" = 'Yes' ]; then
if [ -n "${rgr_opt}" ]; then
if [ -n "${par_opt}" ]; then
for ((clm_idx=${clm_csn_srt_idx};clm_idx<=${clm_csn_end_idx};clm_idx++)); do
wait ${rgr_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR seasonal regrid cmd_rgr[${clm_idx}] failed. Debug this:\n${cmd_rgr[${clm_idx}]}\n"
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_rgr))
echo "Elapsed time to regrid seasonal climos $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
fi # !rgr_opt
fi # !csn_flg
if [ "${ann_flg}" = 'Yes' ]; then
clm_idx=${clm_nbr}
# wait() for annual climatology to finish
if [ -n "${par_opt}" ]; then
wait ${clm_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR annual climo cmd_clm[${clm_idx}] failed. Debug this:\n${cmd_clm[${clm_idx}]}\n"
exit 1
fi # !err
fi # !par_opt
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_clm))
echo "Elapsed time to generate annual climos $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
# Block 5: Regrid climatological annual mean
if [ -n "${rgr_opt}" ]; then
printf "Regrid annual data...\n"
[[ ${dbg_lvl} -ge 1 ]] && date_rgr=$(date +"%s")
#cmd_rgr[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncks -O -t ${thr_nbr} ${nco_opt} ${rgr_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
cmd_rgr[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncremap -u .pid${spt_pid}.climo.${clm_idx}.tmp --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${prc_opt} ${rmp_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
if [ "${mdl_typ}" = 'mpas' ]; then
cmd_rgr[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncremap -C -u .pid${spt_pid}.climo.${clm_idx}.tmp ${prc_opt} ${d2f_opt} ${dpt_opt} --nco_opt=\"${nco_opt}\" -t ${thr_nbr} ${map_opt} ${vrt_opt} ${fl_out[${clm_idx}]} ${fl_rgr[${clm_idx}]}"
fi # !mdl_typ
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_rgr[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
# NB: Do not background climatological mean regridding since it is last step
eval ${cmd_rgr[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR annual regrid cmd_rgr[${clm_idx}] failed. Debug this:\n${cmd_rgr[${clm_idx}]}\n"
exit 1
fi # !err
fi # !dbg
if [ ${dbg_lvl} -ge 1 ]; then
date_crr=$(date +"%s")
date_dff=$((date_crr-date_rgr))
echo "Elapsed time to regrid annual climo $((date_dff/60))m$((date_dff % 60))s"
fi # !dbg
fi # !rgr_opt
fi # !ann_flg
# Link E3SM-climo to AMWG-climo filenames
# drc_pwd is always fully qualified path but drc_out and drc_rgr may be relative paths
# Strategy: Start in drc_pwd, cd to drc_rgr, then link so return code comes from ln not cd
if [ ${lnk_flg} = 'Yes' ]; then
printf "Link E3SM-climo to AMWG-climo filenames...\n"
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
if [ -n "${rgr_opt}" ]; then
cmd_lnk[${clm_idx}]="cd ${drc_pwd};cd ${drc_rgr};ln -s -f ${fl_rgr[${clm_idx}]/${drc_rgr}\//} ${fl_amwg[${clm_idx}]/${drc_rgr}\//}"
else
cmd_lnk[${clm_idx}]="cd ${drc_pwd};cd ${drc_out};ln -s -f ${fl_out[${clm_idx}]/${drc_out}\//} ${fl_amwg[${clm_idx}]/${drc_out}\//}"
fi # !rgr_opt
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_lnk[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
eval ${cmd_lnk[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR linking E3SM to AMWG filename cmd_lnk[${clm_idx}] failed. Debug this:\n${cmd_lnk[${clm_idx}]}\n"
exit 1
fi # !err
fi # !dbg
done # !clm_idx
cd ${drc_pwd}
fi # !lnk_flg
fi # !clm_md !bnr_flg
# Extended climos
if [ "${clm_flg}" = 'Yes' ] && [ "${xtn_flg}" = 'Yes' ]; then
if [ -n "${drc_prv}" ] && [ ! -d "${drc_prv}" ]; then
cmd_mkd="mkdir -p ${drc_prv}"
eval ${cmd_mkd}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR Failed to create previous climatology directory. Debug this:\n${cmd_mkd}\n"
printf "${spt_nm}: HINT Creating a directory requires proper write permissions\n"
exit 1
fi # !err
fi # !drc_prv
if [ -n "${drc_xtn}" ] && [ ! -d "${drc_xtn}" ]; then
cmd_mkd="mkdir -p ${drc_xtn}"
eval ${cmd_mkd}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR Failed to create extended climatology directory. Debug this:\n${cmd_mkd}\n"
printf "${spt_nm}: HINT Creating a directory requires proper write permissions\n"
exit 1
fi # !err
fi # !drc_xtn
trim_leading_zeros ${yr_srt_prv}
yr_srt_rth_prv=${sng_trm}
yyyy_srt_prv=`printf "%04d" ${yr_srt_rth_prv}`
yyyy_clm_srt_dec_prv=${yyyy_srt_prv}
let yr_srtm1_prv=${yr_srt_rth_prv}-1
if [ "${ncr_flg}" = 'Yes' ]; then
let yr_end_prv=${yr_srt_rth}-1
fi # !ncr_flg
trim_leading_zeros ${yr_end_prv}
yr_end_rth_prv=${sng_trm}
yyyy_end_prv=`printf "%04d" ${yr_end_rth_prv}`
let yr_endm1_prv=${yr_end_rth_prv}-1
let yr_nbr_prv=${yr_end_rth_prv}-${yr_srt_rth_prv}+1
let yr_nbr_xtn=${yr_nbr_prv}+${yr_nbr}
wgt_prv=$(echo "${yr_nbr_prv}/${yr_nbr_xtn}" | bc -l)
wgt_crr=$(echo "${yr_nbr}/${yr_nbr_xtn}" | bc -l)
if [ "${bnr_flg}" = 'Yes' ]; then
printf "Produce extended climatology as weighted average of two previously computed climatologies:\n"
else # !bnr_flg
printf "Produce extended climatology as weighted average of previously computed and incremental/new climatologies:\n"
fi # !bnr_flg
# Replace yr_srt by yr_srt_prv in "yrs_averaged" attribute
nco_opt="${nco_opt/${yr_srt}-/${yr_srt_prv}-}"
if [ "${clm_md}" = 'ann' ]; then
printf "Previous/first climatology is ${yr_nbr_prv} years from ${yyyy_srt_prv} to ${yyyy_end_prv}, weight = ${wgt_prv}\n"
printf "Current/second climatology is ${yr_nbr} years from ${yyyy_srt} to ${yyyy_end}, weight = ${wgt_crr}\n"
printf "Extended climatology is ${yr_nbr_xtn} years from ${yyyy_srt_prv} to ${yyyy_end}\n"
fi # !clm_md
if [ "${clm_md}" = 'mth' ]; then
printf "Previous/first climatology is ${yr_nbr_prv} years from ${yyyy_clm_srt_dec_prv}${mm_ann_srt} to ${yyyy_end_prv}${mm_ann_end}, weight = ${wgt_prv}\n"
printf "Current/second climatology is ${yr_nbr} years from ${yyyy_clm_srt_dec}${mm_ann_srt} to ${yyyy_end}${mm_ann_end}, weight = ${wgt_crr}\n"
printf "Extended climatology is ${yr_nbr_xtn} years from ${yyyy_clm_srt_dec_prv}${mm_ann_srt} to ${yyyy_end}${mm_ann_end}\n"
clm_idx=0
for mth in {01..12}; do
let clm_idx=${clm_idx}+1
MM=`printf "%02d" ${clm_idx}`
fl_prv[${clm_idx}]="${drc_prv}/${out_nm}_${MM}_${yyyy_srt_prv}${MM}_${yyyy_end_prv}${MM}_climo.nc"
fl_xtn[${clm_idx}]="${drc_xtn}/${out_nm}_${MM}_${yyyy_srt_prv}${MM}_${yyyy_end}${MM}_climo.nc"
done # !mth
if [ ${wnt_md} = 'djf' ]; then
yyyy_clm_srt_dec_prv=`printf "%04d" ${yr_srtm1_prv}`
yyyy_clm_end_dec_prv=`printf "%04d" ${yr_endm1_prv}`
clm_idx=12
MM=`printf "%02d" ${clm_idx}`
fl_prv[${clm_idx}]="${drc_prv}/${out_nm}_${MM}_${yyyy_clm_srt_dec_prv}${MM}_${yyyy_clm_end_dec_prv}${MM}_climo.nc"
fl_xtn[${clm_idx}]="${drc_xtn}/${out_nm}_${MM}_${yyyy_clm_srt_dec_prv}${MM}_${yyyy_clm_end_dec}${MM}_climo.nc"
fi # !djf
for ((csn_idx=0;csn_idx<${csn_nbr};csn_idx++)); do
# Produce names like "${out_nm}_MAM_${yyyy_srt_prv}03_${yyyy_end_prv}05_climo.nc"
let clm_idx=1+12+${csn_idx}
fl_prv[${clm_idx}]="${drc_prv}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_srt_prv}${csn_srt_mm[${map_r2d[${csn_idx}]}]}_${yyyy_end_prv}${csn_end_mm[${map_r2d[${csn_idx}]}]}_climo.nc"
fl_xtn[${clm_idx}]="${drc_xtn}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_srt_prv}${csn_srt_mm[${map_r2d[${csn_idx}]}]}_${yyyy_end}${csn_end_mm[${map_r2d[${csn_idx}]}]}_climo.nc"
if [ "${csn_abb[${map_r2d[${csn_idx}]}]}" = 'DJF' ]; then
fl_prv[${clm_idx}]="${drc_prv}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec_prv}${mm_djf_srt}_${yyyy_end_prv}${mm_djf_end}_climo.nc"
fl_xtn[${clm_idx}]="${drc_xtn}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec_prv}${mm_djf_srt}_${yyyy_end}${mm_djf_end}_climo.nc"
fi # !DJF
if [ "${csn_abb[${map_r2d[${csn_idx}]}]}" = 'ANN' ]; then
fl_prv[${clm_idx}]="${drc_prv}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec_prv}${mm_ann_srt}_${yyyy_end_prv}${mm_ann_end}_climo.nc"
fl_xtn[${clm_idx}]="${drc_xtn}/${out_nm}_${csn_abb[${map_r2d[${csn_idx}]}]}_${yyyy_clm_srt_dec_prv}${mm_ann_srt}_${yyyy_end}${mm_ann_end}_climo.nc"
fi # !DJF
done # !csn_idx
# fl_prv[13]="${drc_prv}/${out_nm}_MAM_${yyyy_srt_prv}03_${yyyy_end_prv}05_climo.nc"
# fl_prv[14]="${drc_prv}/${out_nm}_JJA_${yyyy_srt_prv}06_${yyyy_end_prv}08_climo.nc"
# fl_prv[15]="${drc_prv}/${out_nm}_SON_${yyyy_srt_prv}09_${yyyy_end_prv}11_climo.nc"
# fl_prv[16]="${drc_prv}/${out_nm}_DJF_${yyyy_clm_srt_dec_prv}${mm_djf_srt}_${yyyy_end_prv}${mm_djf_end}_climo.nc"
# fl_xtn[13]="${drc_xtn}/${out_nm}_MAM_${yyyy_srt_prv}03_${yyyy_end}05_climo.nc"
# fl_xtn[14]="${drc_xtn}/${out_nm}_JJA_${yyyy_srt_prv}06_${yyyy_end}08_climo.nc"
# fl_xtn[15]="${drc_xtn}/${out_nm}_SON_${yyyy_srt_prv}09_${yyyy_end}11_climo.nc"
# fl_xtn[16]="${drc_xtn}/${out_nm}_DJF_${yyyy_clm_srt_dec_prv}${mm_djf_srt}_${yyyy_end}${mm_djf_end}_climo.nc"
if [ "${ann_flg}" = 'Yes' ]; then
fl_prv[${clm_nbr}]="${drc_prv}/${out_nm}_ANN_${yyyy_clm_srt_dec_prv}${mm_ann_srt}_${yyyy_end_prv}${mm_ann_end}_climo.nc"
fl_xtn[${clm_nbr}]="${drc_xtn}/${out_nm}_ANN_${yyyy_clm_srt_dec_prv}${mm_ann_srt}_${yyyy_end}${mm_ann_end}_climo.nc"
fi # !ann_flg
# Derive all regridded and AMWG names from output names
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
fl_rgr_prv[${clm_idx}]="${fl_rgr[${clm_idx}]/${drc_rgr}/${drc_rgr_prv}}"
fl_rgr_prv[${clm_idx}]="${fl_rgr_prv[${clm_idx}]/_${yyyy_srt}/_${yyyy_srt_prv}}"
fl_rgr_prv[${clm_idx}]="${fl_rgr_prv[${clm_idx}]/_${yyyy_end}/_${yyyy_end_prv}}"
fl_rgr_xtn[${clm_idx}]="${fl_rgr[${clm_idx}]/${drc_rgr}/${drc_rgr_xtn}}"
fl_rgr_xtn[${clm_idx}]="${fl_rgr_xtn[${clm_idx}]/_${yyyy_srt}/_${yyyy_srt_prv}}"
fl_amwg_xtn[${clm_idx}]=`expr match "${fl_xtn[${clm_idx}]}" '\(.*\)_.*_.*_climo.nc'` # Prune _YYYYYMM_YYYYMM_climo.nc
fl_amwg_xtn[${clm_idx}]="${fl_amwg[${clm_idx}]}_climo.nc" # Replace with _climo.nc
fl_amwg_xtn[${clm_idx}]="${fl_amwg[${clm_idx}]/${drc_xtn}\//}" # Delete prepended path to ease symlinking
if [ ${wnt_md} = 'djf' ]; then
# Handle Dec, DJF, and ANN
let csn_idx=${clm_idx}-12-1
let ann_idx=${clm_csn_end_idx}+1
if [ ${csn_idx} -ge 0 ]; then
if [ "${csn_abb[${map_r2d[${csn_idx}]}]}" = 'DJF' ] || [ ${clm_idx} -eq ${ann_idx} ]; then
flg_djf_or_ann='Yes'
fi # !djf or ann
fi # !csn_idx
if [ ${clm_idx} -eq 12 ] || [ ${flg_djf_or_ann} = 'Yes' ]; then
fl_rgr_prv[${clm_idx}]="${fl_rgr[${clm_idx}]/${drc_rgr}/${drc_rgr_prv}}"
fl_rgr_prv[${clm_idx}]="${fl_rgr_prv[${clm_idx}]/_${yyyy_clm_srt_dec}/_${yyyy_clm_srt_dec_prv}}"
if [ ${clm_idx} -eq 12 ]; then
fl_rgr_prv[${clm_idx}]="${fl_rgr_prv[${clm_idx}]/_${yyyy_clm_end_dec}/_${yyyy_clm_end_dec_prv}}"
else
fl_rgr_prv[${clm_idx}]="${fl_rgr_prv[${clm_idx}]/_${yyyy_end}/_${yyyy_end_prv}}"
fi # !Dec
fl_rgr_xtn[${clm_idx}]="${fl_rgr[${clm_idx}]/${drc_rgr}/${drc_rgr_xtn}}"
fl_rgr_xtn[${clm_idx}]="${fl_rgr_xtn[${clm_idx}]/_${yyyy_clm_srt_dec}/_${yyyy_clm_srt_dec_prv}}"
fi # !Dec, DJF, ANN
fi # !wnt_md
done # !clm_idx
printf "Weight input climos to produce extended climo...\n"
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
cmd_xtn[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncflint -O ${nco_opt} ${gaa_sng} -w ${wgt_prv},${wgt_crr} ${fl_prv[${clm_idx}]} ${fl_out[${clm_idx}]} ${fl_xtn[${clm_idx}]}"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_xtn[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_xtn[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR extended climo cmd_xtn[${clm_idx}] failed. Debug this:\n${cmd_xtn[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_xtn[${clm_idx}]} ${par_opt} # eval always returns 0 on backgrounded processes
xtn_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
if [ -n "${par_opt}" ]; then
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
wait ${xtn_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR extended climo cmd_xtn[${clm_idx}] failed. Debug this:\n${cmd_xtn[${clm_idx}]}\n"
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
wait
if [ -n "${rgr_opt}" ]; then
printf "Weight input climos to produce extended regridded climo...\n"
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
cmd_rgr_xtn[${clm_idx}]="${cmd_mpi[${clm_idx}]} ncflint -O ${nco_opt} -w ${wgt_prv},${wgt_crr} ${fl_rgr_prv[${clm_idx}]} ${fl_rgr[${clm_idx}]} ${fl_rgr_xtn[${clm_idx}]}"
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_rgr_xtn[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
if [ -z "${par_opt}" ]; then
eval ${cmd_rgr_xtn[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR extended climo cmd_rgr_xtn[${clm_idx}] failed. Debug this:\n${cmd_rgr_xtn[${clm_idx}]}\n"
exit 1
fi # !err
else # !par_opt
eval ${cmd_rgr_xtn[${clm_idx}]} ${par_opt} # eval always returns 0 on backgrounded processes
rgr_xtn_pid[${clm_idx}]=$!
fi # !par_opt
fi # !dbg
done # !clm_idx
if [ -n "${par_opt}" ]; then
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
wait ${rgr_xtn_pid[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR extended climo cmd_rgr_xtn[${clm_idx}] failed. Debug this:\n${cmd_rgr_xtn[${clm_idx}]}\n"
exit 1
fi # !err
done # !clm_idx
fi # !par_opt
wait
fi # !rgr_opt
# Link E3SM-climo to AMWG-climo filenames
# drc_pwd is always fully qualified path but drc_out and drc_rgr may be relative paths
# Strategy: Start in drc_pwd, cd to drc_rgr, then link so return code comes from ln not cd
if [ ${lnk_flg} = 'Yes' ]; then
printf "Link extended E3SM-climo to AMWG-climo filenames...\n"
for ((clm_idx=1;clm_idx<=clm_nbr;clm_idx++)); do
if [ -n "${rgr_opt}" ]; then
cmd_lnk_xtn[${clm_idx}]="cd ${drc_pwd};cd ${drc_rgr_xtn};ln -s -f ${fl_rgr_xtn[${clm_idx}]/${drc_rgr_xtn}\//} ${fl_amwg[${clm_idx}]/${drc_rgr_xtn}\//}"
else
cmd_lnk_xtn[${clm_idx}]="cd ${drc_pwd};cd ${drc_xtn};ln -s -f ${fl_xtn[${clm_idx}]/${drc_xtn}\//} ${fl_amwg[${clm_idx}]/${drc_xtn}\//}"
fi # !rgr_opt
if [ ${dbg_lvl} -ge 1 ]; then
echo ${cmd_lnk_xtn[${clm_idx}]}
fi # !dbg
if [ ${dbg_lvl} -le 1 ]; then
eval ${cmd_lnk_xtn[${clm_idx}]}
if [ "$?" -ne 0 ]; then
printf "${spt_nm}: ERROR linking E3SM to AMWG filename cmd_lnk_xtn[${clm_idx}] failed. Debug this:\n${cmd_lnk_xtn[${clm_idx}]}\n"
exit 1
fi # !err
fi # !dbg
done # !clm_idx
cd ${drc_pwd}
fi # !lnk_flg
fi # !clm_md
else # !xtn_flg extended climos
yr_nbr_xtn=${yr_nbr}
fi # !xtn_flg extended climos
if [ "${clm_flg}" = 'Yes' ]; then
if [ "${clm_md}" = 'dly' ]; then
echo "Quick plots of last climatological daily mean:"
let idx_lst=${clm_nbr}-1
else
echo "Quick plots of last computed climatology:"
let idx_lst=${clm_nbr}
fi # !dly
if [ -n "${yr_srt_prv}" ]; then
if [ -n "${rgr_opt}" ]; then
echo "ncvis ${fl_rgr_xtn[${idx_lst}]} &"
else
echo "ncvis ${fl_xtn[${idx_lst}]} &"
fi # !rgr_opt
else
if [ -n "${rgr_opt}" ]; then
echo "ncvis ${fl_rgr[${idx_lst}]} &"
else
echo "ncvis ${fl_out[${idx_lst}]} &"
fi # !rgr_opt
fi # !yr_srt_prv
fi # !clm_flg
if [ "${tms_flg}" = 'Yes' ]; then
echo "Quick plots of last timeseries segment of last variable split:"
let idx_lst=${var_nbr}-1
if [ -n "${rgr_opt}" ] && [ ${no_ntv_tms} = 'No' ]; then
echo "ncvis ${fl_rgr[${idx_lst}]} &"
else
echo "ncvis ${fl_out[${idx_lst}]} &"
fi # !rgr_opt
fi # !tms_flg
date_end=$(date +"%s")
if [ -n "${caseid}" ]; then
printf "Completed ${yr_nbr_xtn}-year climatology operations for dataset with caseid = ${caseid} at `date`\n"
else # !caseid
printf "Completed ${yr_nbr_xtn}-year climatology operations for input data at `date`\n"
fi # !caseid
date_dff=$((date_end-date_srt))
echo "Elapsed time $((date_dff/60))m$((date_dff % 60))s"
exit 0
|