1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 3747 3748 3749 3750 3751 3752 3753 3754 3755 3756 3757 3758 3759 3760 3761 3762 3763 3764 3765 3766 3767 3768 3769 3770 3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 4066 4067 4068 4069 4070 4071 4072 4073 4074 4075 4076 4077 4078 4079 4080 4081 4082 4083 4084 4085 4086 4087 4088 4089 4090 4091 4092 4093 4094 4095 4096 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 4116 4117 4118 4119 4120 4121 4122 4123 4124 4125 4126 4127 4128 4129 4130 4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 4152 4153 4154 4155 4156 4157 4158 4159 4160 4161 4162 4163 4164 4165 4166 4167 4168 4169 4170 4171 4172 4173 4174 4175 4176 4177 4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 4190 4191 4192 4193 4194 4195 4196 4197 4198 4199 4200 4201 4202 4203 4204 4205 4206 4207 4208 4209 4210 4211 4212 4213 4214 4215 4216 4217 4218 4219 4220 4221 4222 4223 4224 4225 4226 4227 4228 4229 4230 4231 4232 4233 4234 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 4250 4251 4252 4253 4254 4255 4256 4257 4258 4259 4260 4261 4262 4263 4264 4265 4266 4267 4268 4269 4270 4271 4272 4273 4274 4275 4276 4277 4278 4279 4280 4281 4282 4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 4300 4301 4302 4303 4304 4305 4306 4307 4308 4309 4310 4311 4312 4313 4314 4315 4316 4317 4318 4319 4320 4321 4322 4323 4324 4325 4326 4327 4328 4329 4330 4331 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 4353 4354 4355 4356 4357 4358 4359 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369 4370 4371 4372 4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 4413 4414 4415 4416 4417 4418 4419 4420 4421 4422 4423 4424 4425 4426 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 4506 4507 4508 4509 4510 4511 4512 4513 4514 4515 4516 4517 4518 4519 4520 4521 4522 4523 4524 4525 4526 4527 4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 4545 4546 4547 4548 4549 4550 4551 4552 4553 4554 4555 4556 4557 4558 4559 4560 4561 4562 4563 4564 4565 4566 4567 4568 4569 4570 4571 4572 4573 4574 4575 4576 4577 4578 4579 4580 4581 4582 4583 4584 4585 4586 4587 4588 4589 4590 4591 4592 4593 4594 4595 4596 4597 4598 4599 4600 4601 4602 4603 4604 4605 4606 4607 4608 4609 4610 4611 4612 4613 4614 4615 4616 4617 4618 4619 4620 4621 4622 4623 4624 4625 4626 4627 4628 4629 4630 4631 4632 4633 4634 4635 4636 4637 4638 4639 4640 4641 4642 4643 4644 4645 4646 4647 4648 4649 4650 4651 4652 4653 4654 4655 4656 4657 4658 4659 4660 4661 4662 4663 4664 4665 4666 4667 4668 4669 4670 4671 4672 4673 4674 4675 4676 4677 4678 4679 4680 4681 4682 4683 4684 4685 4686 4687 4688 4689 4690 4691 4692 4693 4694 4695 4696 4697 4698 4699 4700 4701 4702 4703 4704 4705 4706 4707 4708 4709 4710 4711 4712 4713 4714 4715 4716 4717 4718 4719 4720 4721 4722 4723 4724 4725 4726 4727 4728 4729 4730 4731 4732 4733 4734 4735 4736 4737 4738 4739 4740 4741 4742 4743 4744 4745 4746 4747 4748 4749 4750 4751 4752 4753 4754 4755 4756 4757 4758 4759 4760 4761 4762 4763 4764 4765 4766 4767 4768 4769 4770 4771 4772 4773 4774 4775 4776 4777 4778 4779 4780 4781 4782 4783 4784 4785 4786 4787 4788 4789 4790 4791 4792 4793 4794 4795 4796 4797 4798 4799 4800 4801 4802 4803 4804 4805 4806 4807 4808 4809 4810 4811 4812 4813 4814 4815 4816 4817 4818 4819 4820 4821 4822 4823 4824 4825 4826 4827 4828 4829 4830 4831 4832 4833 4834 4835 4836 4837 4838 4839 4840 4841 4842 4843 4844 4845 4846 4847 4848 4849 4850 4851 4852 4853 4854 4855 4856 4857 4858 4859 4860 4861 4862 4863 4864 4865 4866 4867 4868 4869 4870 4871 4872 4873 4874 4875 4876 4877 4878 4879 4880 4881 4882 4883 4884 4885 4886 4887 4888 4889 4890 4891 4892 4893 4894 4895 4896 4897 4898 4899 4900 4901 4902 4903 4904 4905 4906 4907 4908 4909 4910 4911 4912 4913 4914 4915 4916 4917 4918 4919 4920 4921 4922 4923 4924 4925 4926 4927 4928 4929 4930 4931 4932 4933 4934 4935 4936 4937 4938 4939 4940 4941 4942 4943 4944 4945 4946 4947 4948 4949 4950 4951 4952 4953 4954 4955 4956 4957 4958 4959 4960 4961 4962 4963 4964 4965 4966 4967 4968 4969 4970 4971 4972 4973 4974 4975 4976 4977 4978 4979 4980 4981 4982 4983 4984 4985 4986 4987 4988 4989 4990 4991 4992 4993 4994 4995 4996 4997 4998 4999 5000 5001 5002 5003 5004 5005 5006 5007 5008 5009 5010 5011 5012 5013 5014 5015 5016 5017 5018 5019 5020 5021 5022 5023 5024 5025 5026 5027 5028 5029 5030 5031 5032 5033 5034 5035 5036 5037 5038 5039 5040 5041 5042 5043 5044 5045 5046 5047 5048 5049 5050 5051 5052 5053 5054 5055 5056 5057 5058 5059 5060 5061 5062 5063 5064 5065 5066 5067 5068 5069 5070 5071 5072 5073 5074 5075 5076 5077 5078 5079 5080 5081 5082 5083 5084 5085 5086 5087 5088 5089 5090 5091 5092 5093 5094 5095 5096 5097 5098 5099 5100 5101 5102 5103 5104 5105 5106 5107 5108 5109 5110 5111 5112 5113 5114 5115 5116 5117 5118 5119 5120 5121 5122 5123 5124 5125 5126 5127 5128 5129 5130 5131 5132 5133 5134 5135 5136 5137 5138 5139 5140 5141 5142 5143 5144 5145 5146 5147 5148 5149 5150 5151 5152 5153 5154 5155 5156 5157 5158 5159 5160 5161 5162 5163 5164 5165 5166 5167 5168 5169 5170 5171 5172 5173 5174 5175 5176 5177 5178 5179 5180 5181 5182 5183 5184 5185 5186 5187 5188 5189 5190 5191 5192 5193 5194 5195 5196 5197 5198 5199 5200 5201 5202 5203 5204 5205 5206 5207 5208 5209 5210 5211 5212 5213 5214 5215 5216 5217 5218 5219 5220 5221 5222 5223 5224 5225 5226 5227 5228 5229 5230 5231 5232 5233 5234 5235 5236 5237 5238 5239 5240 5241 5242 5243 5244 5245 5246 5247 5248 5249 5250 5251 5252 5253 5254 5255 5256 5257 5258 5259 5260 5261 5262 5263 5264 5265 5266 5267 5268 5269 5270 5271 5272 5273 5274 5275 5276 5277 5278 5279 5280 5281 5282 5283 5284 5285 5286 5287 5288 5289 5290 5291 5292 5293 5294 5295 5296 5297 5298 5299 5300 5301 5302 5303 5304 5305 5306 5307 5308 5309 5310 5311 5312 5313 5314 5315 5316 5317 5318 5319 5320 5321 5322 5323 5324 5325 5326 5327 5328 5329 5330 5331 5332 5333 5334 5335 5336 5337 5338 5339 5340 5341 5342 5343 5344 5345 5346 5347 5348 5349 5350 5351 5352 5353 5354 5355 5356 5357 5358 5359 5360 5361 5362 5363 5364 5365 5366 5367 5368 5369 5370 5371 5372 5373 5374 5375 5376 5377 5378 5379 5380 5381 5382 5383 5384 5385 5386 5387 5388 5389 5390 5391 5392 5393 5394 5395 5396 5397 5398 5399 5400 5401 5402 5403 5404 5405 5406 5407 5408 5409 5410 5411 5412 5413 5414 5415 5416 5417 5418 5419
|
GNU Astronomy Utilities NEWS -*- outline -*-
Copyright (C) 2015-2025 Free Software Foundation, Inc.
See the end of the file for license conditions.
* Noteworthy changes in release 0.24 (library 22.0.0) (2025-11-26)
** New publications
- https://arxiv.org/abs/2510.12940v1 by Eskandarlou et al. that describes
the latest status of our scripts to extract and subtract the extended
PSF of bright stars in the J-PLUS survey. See Figure 1 (for an outline
of the flow chart) and Figure 6 (on a nice demo of the effect). It
builds upon Infante-Sainz et al. 2020 (which was the basis for
Gnuastro's 'astscript-psf-*' scripts) and like that paper, this one is
also fully reproducible so you can go inside its public source code to
check every step (see Section 6 for the links).
** New features
*** Arithmetic
- New Operators: similar operators are grouped; groups are sorted
alphabetically by first operator in the group
- filter-minimum:
- filter-maximum:
- filter-sigclip-std:
- filter-madclip-mad:
- filter-sigclip-std:
- filter-madclip-mad:
Replace each pixel with the respective statistic of the pixels in the
given filter size. These are useful when you need to measure the
spread of nearby pixels (in the presence of outliers).
- label-area:
- label-minimum:
- label-maximum:
Given a labeled and a values image do the requested measurements on
the label and set the label's pixels to the output of the measurement
in the output. For example, if you want to remove all labels with an
area less than 3, you can use the 'label-area' option followed by the
'gt' (greater-than) operator. Until now, to do this, we needed to run
MakeCatalog, extract the rows, and write loops to remove the labels
one by one (which was slow, in efficient and could by buggy). These
new operators greatly simply such operations while significantly
improving the speed. See example usages within the book.
- luminosity-to-mag:
- mag-to-luminosity:
Convert apparent magnitude to luminosity and vice-versa. Implemented
by Sepideh Eskandarlou.
- madclip-all:
- sigclip-all:
Return all available statistics after clipping outliers (mean, std,
median and MAD and number of used datasets).
- sblim-diff: calculate the expected difference in surface brightness
limit (purely based on exposed area (of the telescope) and exposure
time (assuming the same filter, sky conditions and assuming no
correlated noise). This can be used for planning projects with
different telescopes, but be careful about filters and atmosphere.
*** ConvertType
--cmapblankcolor: Name of color to use for blank values when showing
single-channel images through a colorbar. This is very useful in
scenarios where you don't want your audience to confuse NaN pixels with
pixels that have data. For example if you use the Viridis colormap, you
can set '--cmapblankcolor=white' or '--cmapblankcolor=red' (which do
not exist in the Viridis color map). Note that this option does not
apply to the "gray" colormap because that is single channel. To get the
list of recognized colors in Gnuastro run 'astconvertt
--listcolors'. This was suggested by Sepideh Eskandarlou.
--cmappgfplots: create a second output file containing the PGFPlots
colormap definition for single-channel data that are to be displayed
through a colormap. PGFPlots is a powerful plot creation package within
LaTeX to create high-quality plots and figures in your papers, slides
or reports. With this option, you can use Gnuastro colormaps that are
not available in PGFPlots. The "Annotations for figure in paper"
section of the book has a fully working example of the usage of
PGFPlots in combination with Gnuastro for high-quality figure
generation.
*** MakeCatalog
--novalinerror: ignore pixel in the values image when calculating the
variance of each label's pixel: only use the pixel's value in the
'--instd' image (which can also be the variance with the '--variance'
option). This is useful in scenarios where the STD/variance image is
not just the sky STD/Variance, but also has the signal's contribution.
--mean-error: Measure the error of the mean for each label (object or
clump).
--meta-measures: new option to print metameasurements (measurements on
measurements) as keywords in the 0th HDU of the output. These include
the measured surface brightness limit, the noise-based magnitude limit
and the confusion limit. Until now, only the first was always
automatically measured, but from this version, if you need to call this
option to get it (the keyword names have also changed, see "MakeCatalog
output keywords" section of the book). For more, see the newly added
section in the book called "Metameasurements on full input".
--nml-sigma and --nml-area: the multiple of standard deviation and area
(in square arcseconds) to use for the noise-based magnitude limit (also
called "detection limit" by some authors).
--cnl-with-objects: measure the confusion limit using objects instead of
clumps (only recommended if you are not generating your labels with
Segment).
--cnl-check: add an extra HDU to the output that contains the distances
to the nearest neighbor of each row used in the confusion limit.
*** Match
- Ambiguous matches (multiple counterparts in other catalog within
aperture) will not be present in the main output. When present, the
ambiguous rows of the second input will be listed in a dedicated HDU of
the output (called 'FLAGGED-IN-2ND'). A dedicated new section (called
"Unambiguous matching") has been added to the book that describes the
intricacies of this new behavior.
--arrange: the arrangement of the match output. Until now, the Match
program only had the 'inner' arrangement (known in SQL as "INNER
JOIN"). From this version, other arrangements are also available and
fully described (with example tables and short tutorials) in the newly
added "Arranging match output" section of the book. A short summary of
the new arrangements is given below:
- full: known as "FULL OUTER JOIN" in SQL. Its output has both the
matched rows and the non-matched rows in one catalog.
- outer: known as "RIGHT OUTER JOIN" in SQL. Its output has the same
number of rows as the second (query) input catalog to Match, giving
the ability to find the (possibly repeated) matching rows from the
first (reference) catalog. See the demo for a real-world example.
- outer-within-aperture: similar to 'outer', but if the nearest
reference catalog entry is more distant than the given aperture, it
will be NaN/blank in the output.
*** Statistics
- Polynomial fitting is now possible for 2D inputs: given either as an
image (assuming the pixel's horizontal and vertical positions are the
independent variables) or as a three columns (where the first two are
independent variables). A new "Two dimensional polynomial fitting"
section has been added in the book to demonstrate its usage.
*** astscript-ds9-region
--fontsize: new option to allow setting the font size of the values
within the name column (given to '--namecol'). The values of the name
column are printed as text on top of each region in DS9.
*** astscript-fits-view
--topcat4k (or '-k') option is added for easy display on 4K monitors
(where the TOPCAT window can be very small by default).
*** astscript-radial-profile
--azimuth option can be called multiple times so the profile is generated
over discrete azimuthal regions of the input. For example if called
with '--azimuth=30,50 --azimuth=210,250', the profile (or polar plot if
requested) will be generated only using pixels within these two
azimuthal ranges (all other pixels are ignored). Suggested and
implemented by Ignacio Ruiz Cejudo.
*** Makefile extensions
- $(ast-text-next TARGET, LIST): returns the next word (after TARGET) in
the input LIST; see the documentation for more.
- $(ast-text-next-words TARGET, NUM, LIST): returns the next NUM words
after TARGET from LIST.
*** Library
- gal_data_append_second_array_to_first_free: for easily appending
(adding the contents of the second array into the first) in
single-dimensional datasets.
- gal_dimension_image_to_table: function to extract all non-blank
elements of an image and save them into a table with their pixel
coordinates (first two columns) and pixel value.
- GAL_FIT_MATRIX_INVALID:
- GAL_FIT_MATRIX_POLYNOMIAL_1D:
- GAL_FIT_MATRIX_POLYNOMIAL_2D:
- GAL_FIT_MATRIX_POLYNOMIAL_2D_TPV:
Macros to identify different types of fitting matrices.
- GAL_FIT_MATRIX_NUMBER_1D:
- GAL_FIT_MATRIX_NUMBER_ALL:
The number of 1D matrices and total matrices used in fitting.
- gal_fit_polynomial_tikhonov: new function to perform a regularized fit
using Tikhonov regression, wrapping over the GNU Scientific Library
(GSL) functions.
- gal_kdtree_range: returns the list of points that are within a given
radius of the given point. Implemented by Barış Güngör.
- gal_label_measure: Replace a label with measured values over it (see
description of newly added 'label-*' operators in Arithmetic).
- gal_list_sizetsizetf64_add:
- gal_list_sizetsizetf64_pop:
- gal_list_sizetsizetf64_free:
New functions to use a new list type that includes a 'size_t' and a
'double'.
- gal_statistics_minimums_three: return the three smallest values of the
given dataset.
- gal_statistics_range_double: return the difference between the minimum
and maximum as a double-precision floating point of a dataset (with any
type).
- gal_tile_per_label: create an array/list of tiles that cover the
separate labels within the input.
- gal_table_sort: sort the given table by a given column.
- gal_units_sblim_diff: calculate the difference in surface brightness
limit only assuming a different effective radius of a different
telescope and/or a different exposure time.
- gal_units_mag_to_luminosity:
- gal_units_luminosity_to_mag:
Convert the observed magnitude of an object to luminosity and
vice-versa.
- gal_units_wavelength_flux_density_to_jy: convert given wavelength flux
density (erg/s/cm^2/A) to Janskys.
** Removed features
- Nothing has been removed in this release.
** Changed features
*** Book
- The term "stack" was used in two contexts within the Arithmetic
program: referring both to the stack of operands during its operation
and the coadding process (when we add many images to make a deeper
one). When talking about the latter scenario, the book now uses the
term "Coadd" (which is also commonly used in the optical astronomical
community).
- The sub-sections of MakeCatalog's old "Quantifying measurement limits"
section (which described individual source errors as well as
metameasures over the whole image) have been distributed in other/new
sub-sections following MakeCatalog's new approach to metameasures as
described above in the new '--meta-measures' option.
*** All programs
--cite: also prints the citation for the Gnuastro book/manual (using the
Zenodo DOI for that release).
*** Arithmetic
- Operators:
- interpolate-minofregion:
- interpolate-maxofregion:
Now take three operands (in order of popping): the outer
connectivity, the inner connectivity and the image with blank
pixels. Until now, it took only the inner connectivity and the outer
connectivity was hard-coded to the maximum of the input's
dimensions. This was necessary because it can happen that the most
permissive outer connectivity may be too loose in some usage
scenarios. To get the same output as before in your existing scripts,
add a '2' before the operator (for an image) or (3 for a cube).
- madclip-*: when the MAD is zero (median is repeated; mostly occurs on
integer inputs), an alternative to the MAD will be used (difference
between the two differing values just before and after the median in
a sorted array). This greatly helps the usability of MAD-clipping on
integer datasets with small scatter (where the median can be repeated
a lot).
--append: when called with this option, Arithmetic will not write its
run-time option values in the 0-th HDU.
--metaname: can now accept multiple values (names of each HDU for
multi-output calls of Arithmetic).
--metaunit: similar to '--metaname'
--metacomment: similar to '--metaname'
*** Crop
- The log-file name (created when run with '--log') is based on the
output file name (see the description of '--log' under the Crop program
for the details) and is a FITS table with a suffix '-log.fits'. Until
now, the log file was always called 'astcrop.log' and it was created in
the running directory. This would cause problems for parallel
executions. Also, it was a plain-text file which is less efficient:
slower to read and write and would be larger for longer catalog inputs.
*** MakeCatalog
- The output should always be a FITS file. This was necessary because
with the metameasures, more and more tables may be requested by the
user. Also, keywords are becoming more important and and supporting
both text and FITS was becoming a major development overhead because
plain-text doesn't support multiple tables per file or keywords.
- All the metadata keywords of all the tables are now written in the 0th
HDU. Until now, they were written in all the produced tables that was
inefficient and confusing; this is part of an already started project
for all Gnuastro programs to write all their metadata in the 0th
HDU. The new "MakeCatalog output keywords" section in the book fully
describes all the output keywords, please check that section for all
the new keyword names. The most significant is the surface brightness
limit keyword(s), that have now been distributed and renamed to fit the
new metameasure structure.
--sbl-sigma: new name for the old '--sfmagnsigma', as part of the process
to standardize metameasures.
--sbl-area: new name for the old '--sfmagarea', as part of the process to
standardize metameasures.
--forcereadstd has been removed in favor of the more general
'--meta-measure' option. This is because it was only useful for
calculating the surface brightness limit (which is a metameasure).
*** Match
- Ambiguous matches (when more than one object is within the given
aperture) are flagged and kept in a new HDU of the output. A new
"Unambiguous matching" section has been added in the "Match" section of
the book to describe this important feature.
- Output can only be in FITS format. Until this version, the output could
be plain-text also. However, plain-text tables can only be a single
table, so multiple files where necessary when multiple tables were
created. Managing all the files in the various run-time scenarios was
causing a significant slow-down of the development and was also bugging
(for example '--logasoutput' which was not thread-safe). If a
plain-text format is necessary in your pipelines, you can call
Gnuastro's Table program to convert the given HDU into plain-text.
--outcols: can now be invoked multiple times in one command. Until this
version it was only possible to call it once, causing inconveniently
long options for large tables. This was suggested by Sepideh
Eskandarlou.
*** astscript-ds9-region
- The default coordinate system to write WCS coordinates for DS9 is now
ICRS. Until now, it was FK5 (based on the default system that DS9 would
propose for writing regions). But DS9's latest versions have changed to
ICRS, so we are also doing the same here. In practice, the difference
is very small and only relevant for extremely precise positioning.
*** astscript-pointing-simulate
--coadd-operator: new name for '--stack-operator'.
*** astscript-zeropoint
- The default matching radius for stars has been changed to 0.5
arcseconds. Until this version it was 0.2 arcseconds.
- If less than three stars are matched between the reference and query
catalogs, the script will abort with an elaborate error message
(guiding you on how to fix the problem. This is because the result will
not statistically reliable.
*** Library
- gal_blank_remove: new 'free_if_all_blank' to enable optional freeing of
the input array when all elements are blank.
- gal_kdtree_nearest_neighbor: new name of 'gal_kdtree_nearest_neighbour'
which was in British English (Gnuastro uses American English).
- gal_match_kdtree: a new 'arrange' and 'flag' argument have been added
to define the arrangement of the output.
- gal_match_sort_based: a new 'flag' argument has been added that will be
filled if there are ambiguous matches.
- gal_table_write_log: a new 'format' argument is added to allow setting
the format of the table (until now, it was always plain-text).
- New names for the fitting library functions (old --> new). This is
because (as described in the new features), the fitting functions have
generalized to also accept 2D inputs.
- gal_fit_1d_linear --> gal_fit_linear_1d
- gal_fit_1d_polynomial --> gal_fit_polynomial
- gal_fit_1d_polynomial_robust --> gal_fit_polynomial_robust
- gal_fit_1d_linear_estimate --> gal_fit_linear_no_constant_1d
- gal_fit_1d_linear_no_constant --> gal_fit_linear_no_constant_1d
** Bugs fixed
- bug #66043: One pixel offset in PSF subtraction.
- bug #66216: NaN values in MakeProfile outputs for very small profiles
(smaller than a pixel even after accounting for the truncation
factor). Reported by Alejandro Camazón Pinilla.
- bug #66262: color-faint-gray crashes when using --graykernelfwhm. Found
and fixed by Samane Raji.
- bug #66280: Convolve's check images not in same directory as output.
- bug #66303: astscript-color-faint-gray crashes if using 4th
channel. Reported by Elizabeth Sola and fixed by Raul Infante-Sainz.
- bug #66308: Library's 'gal_wcs_img_to_world' and 'gal_wcs_world_to_img'
returned inverse names when the operation was not in-place. Found and
fixed by Giacomo Lorenzetti.
- bug #66309: Match crashes if kdtree is not set. Found and fixed by
Giacomo Lorenzetti.
- bug #66572: MakeProfiles continues adding pixels after truncation when
the profile is small (a couple of pixels), and the gradient is strong.
- bug #66588: Arithmetic's 'trim' operator does not produce 0x0 image
when all the image is blank.
- bug #66657: gal_fits_img_read not passing hdu_option_name; found with
the help of Sepideh Eskandarlou.
- bug #66693: MakeProfile crashes with segmentation fault with when
--customimg is given and the oversampling factor is larger than 1.
- bug #66867: Example added in book to show how to remove outputs of
Query's '--overlapwith' option when sources outside of the image are
also retrieved (for example due to a rotated image or the server not
accounting for spherical distortion). Reported by Raul Infante-Sainz.
- bug #66869: Crop segmentation fault leading to unpredictable return
values. Reported by Raul Infante-Sainz.
- bug #67105: astscript-radial-profile produced unreasonable output when
undersample was given as one. Reported by Samane Raji and fixed by Raul
Infante-Sainz.
- bug #67106: Crash during build when Ghostscript is not
present. Reported by Faezeh Bidjarchian.
- #67128: Maskfilled operators wrong output with zero scatter (can happen
in integer images, for example in the creation of master dark frames).
- bug #67190: Sorting an empty plain-text table gives segmentation
fault. Reported by Sepideh Eskandarlou.
- bug #67191: Arithmetic does not parse '-inf' as a number on the
command-line and complains with an invalid '-i' option! Reported by
Sepideh Eskandarlou.
- bug #67204: Table's --equal operator cannot be called multiple times on
same column with different values.
- bug #67232: Building failure on i686 architectures (32-bit). Reported
by Benson Muite and fixed by Giacomo Lorenzetti.
- bug #67275: Segmentation fault in MakeCatalog during clump catalog
generation when input labels not contiguous (for example you are
feeding a crop of a larger object/clump map).
- bug #67297: Fits program segmentation fault when no input specified and
a HDU operation requested.
- bug #67364: Match's k-d tree based inner matching depends on order of
inputs. Fixed by Barış Güngör.
- bug #67475: Statistics fitting output prints 0 when the values are very
small. Reported by Faezeh Bidjarchian.
- bug #67684: Match's full-arrangement with outcols produces an empty
output when there is no match. Reported by Sepideh Eskandarlou.
* Noteworthy changes in release 0.23 (library 21.0.0) (2024-07-13)
** New publications
- https://ui.adsabs.harvard.edu/abs/2024RNAAS...8..168E by Eskandarlou
and Akhlaghi describing the newly added polar plot capability of
'astscript-radial-profile' (see the figure in the paper).
** New features
*** Arithmetic
--append: if the output file already exists, don't delete it, add the
outputs as extra HDUs (by default, the output file is deleted). If the
output does not exist, then this option has no affect.
- New operators (also available in "Column Arithmetic" of the Table
program):
- jy-to-wavelength-flux-density: converts Janskys to wavelength flux
density (erg/cm^2/s/Ã…) at the given wavelength.
- filter-madclip-mean: filter/smooth the input using MAD-clipped mean.
- filter-madclip-median: filter/smooth the input using MAD-clipped
median.
- free: free (from memory) the top operand on the stack of
operands. This is useful in combination with operators that produce
more than one output operand.
- madclip-maskfilled: mask (set to NaN) all input elements that are
outliers (defined by MAD-clipping). Combined with the stacking
operators this allows removing large contiguous outliers in your
final stacked datasets.
- siglclip-maskfilled: similar to 'madclip-maskfilled', but defining
outliers by Sigma-clipping.
- zeropoint-change: change the zero point of the input data set to a
new zero point.
*** Statistics
--concentration: measure the "concentration" of values in a distribution
around the median. See the book for the full description of this
option.
*** astscript-fits-view
--globalhdu: use the same HDU in any number of input files (with the
short format of '-g'); similar to the same option in Arithmetic or
ConvertType.
--ds9region: load any number of DS9 region files (separated by comas and
given to this option) into the DS9 window that will be opened. This was
added by Giacomo Lorenzetti.
- The default DS9 'sls' colormap contrast and bias are adjusted so the
largest value does not have a white color, but is dark red. This was
done because white is DS9's default color for blank pixels and
out-of-image regions; making the pixels with the largest value
invisible.
*** astscript-radial-profile
--polar: generate a 2D polar plot in the same region of the image that
the 1D radial profile has been calculate over. The polar plot is a
mapping/projection of pixels into an azimuthal angle vs. radius grid,
highlighting features that have polar symmetry (for example spiral arms
in galaxies or diffraction spikes in the PSF). This feature was
implemented by Sepideh Eskdandarlou.
*** astscript-psf-scale-factor
--nocentering: disable sub-pixel centering of the desired star before
finding the scale factor; added by Sepideh Eskandarlou and Raul
Infante-Sainz.
*** Makefile extensions
- $(ast-text-prev TARGET, LIST): select the word that is previous to
'TARGET' in a list of words. See the minimal working example in the
book for more.
- $(ast-text-prev-batch TARGET, NUM, LIST): select the previous "batch"
of 'NUM' words (in relation to the batch that contains 'TARGET'). This
is useful for steps in your pipelines were you need to limit the
parallelization to batches. See the minimal working example in the book
for more.
- $(ast-text-prev-batch-by-ram TARGET, NEEDED_RAM_GB, LIST): select the
previous batch of words in 'LIST' such that the total consumed RAM by
all the parallel executions does not exceed the available RAM when Make
starts. Like 'ast-text-prev-batch', this is useful for steps in a
pipeline that require a large amount of RAM (thus not allowing parallel
execution), but this is more generic and adapts to different systems
with very different RAM and/or CPU threads. See the minimal working
example in the book for more.
*** Library
- gal_statistics_concentration: measure the concentration of values
around the median; see the book for the details.
- gal_units_jy_to_wavelength_flux_density: convert Janskys to wavelength
flux density.
- gal_units_zeropoint_change: change the zero point of the input data set
to an output zero point.
** Removed features
** Changed features
*** All programs
- System configuration files are installed in 'PREFIX/etc/gnuastro/',
('PREFIX/' is the installation directory of Gnuastro; when installing
from source, and if no '--prefix' is given to the './configure script,
it is '/usr/local'). Until now, the configuration files of Gnuastro's
programs were installed in 'PREFIX/etc/'. But that was a top-level
directory for the configuration files of all the programs within the
operating system. Therefore the old method would unnecessarily populate
that already crowded directory, and finding Gnuastro's configuration
files could become non-trivial. This was suggested by Phil Wyett.
- User configuration files are installed in '$HOME/.local/etc/gnuastro/'
instead of '$HOME/.local/etc/'. See the description of system
configuration files above for more.
*** Arithmetic
- The following operators will output two operands: the main statistic
and the number of inputs used in each pixel: 'sigclip-mean',
'sigclip-median', 'sigclip-std', 'sigclip-mad', 'madclip-mean',
'madclip-median', 'madclip-std', 'madclip-mad'. See the book for full
examples on how to work with this (in summary: use '--writeall' to also
write the numbers image in the output or use 'swap free' to free it).
- Removed operators:
- 'madclip-number' and 'sigclip-number': because the numbers image will be
returned with any clipping stacking operator.
- 'madclip-fill-mad', 'sigclip-fill-mad', 'madclip-fill-std',
'sigclip-fill-std', 'madclip-fill-mean', 'sigclip-fill-mean',
'madclip-fill-median', 'sigclip-fill-median', 'madclip-fill-std',
'sigclip-fill-std': because the newly added and more general
operators ('madclip-maskfilled' and 'sigclip-maskfilled') can be
combined with any of the stacking operators to produce these as well
as many other useful scenarios.
*** ConvertType
- Standard input (stdin) is only checked when no input file has been
specified. Until now, the 'stdin' would be checked all the time, and if
anything was present there, it would be taken as the first color
channel. But this could cause problems in shell loops that parse the
standard input (for example, see
https://lists.gnu.org/archive/html/help-bash/2024-02/msg00074.html). Also,
it would rarely happen that when the three RGB color channels are
necessary, the first would be from the standard input, and the other
two as files. The new behavior is much more similar to the behavior of
other command-line programs like AWK or SED. This issue was reported by
Jesús Vega and Raul Infante-Sainz and solved with the help of Greg
Wooledge and Dennis Williamson.
*** CosmicCalculator
- The names of spectral lines which include their rest frame wavelength
is rounded to the nearest integer. For example the OIII line at
wavelength of 5006.84 Angstroms is now called O-III-5007. Until now,it
was confusingly/wrongly called O-III-5006!
*** Library
- speclines.h: macros specifying spectral lines have been corrected to
round wavelengths to nearest integer (see description of changes in
CosmicCalculator). For example the old 'GAL_SPECLINES_O_III_5006' is
now called 'GAL_SPECLINES_O_III_5007'.
*** astscript-fits-view
- The short format of the '--ds9geometry' option is '-G' (until now it
was '-g'). This was necessary to allow the '-g' of this script to have
a similar behavior to the other Gnuastro programs that take multiple
inputs (like Arithmetic or ConvertType) and '-g' is short for
'--globalhdu' (so the same HDU is opened in all the inputs).
** Bugs fixed
- bug #65255: description of CosmicCalculator's '--arcsectandist' didn't
specify if it is in physical or comoving coordinates. Found and fixed
by Boud Roukema.
- bug #65267: 'astscript-fits-view' created incorrect call to ds9 when
the number of input files was more than the specified HDUs.
- bug #65269: 'astscript-color-faint-gray' and 'astscript-psf-stamp' use
a coma as the decimal point on some operating systems. Reported by
Jesús Vega and fixed by Raul Infante-Sainz.
- bug #65377: MakeCatalog's suggested command when labeled image has
negative values (and MakeCatalog crashes) is wrong. Reported by Zahra
Sharbaf.
- bug #65382: Radial profile script truncates the azimuthal angle when
the maximum requested azimuthal range is above 360 degrees. Reported by
Fernando Buitrago Alonso and fixed by Raul Infante-Sainz.
- bug #63064: Statistics crashes when all pixels are NaN. Reported by
Raul Infante-Sainz and fixed with help of Thorsten Alteholz.
- bug #65494: Crop segmentation fault with older CFITSIOs (three word
version format). Reported by Giacomo Lorenzetti.
- bug #65571: Crop returns successfully when no output was made. Reported
by Sepideh Eskandarlou.
- bug #65561: Projections to infinity cause very long running time in
Warp. Found and fixed by Giacomo Lorenzetti.
- bug #63026: Warp translation adds one extra pixel even if it is not
requested. Reported by Raul Infante-Sainz and fixed by Giacomo
Lorenzetti.
- bug #65672: ./developer-build exits silently when some options are
given an empty value. Found and fixed by Giacomo Lorenzetti.
- bug #65673: Arithmetic creates a metadata FITS file a HDU with no data)
even when the output is not FITS (plain-text or on the standard
output). Reported by Sepideh Eskandarlou.
- bug #65743: Arithmetic's collapse-number mistakenly calculating
collapse-median instead. Reported by Sepideh Eskandarlou.
- bug #65801: astscript-fits-view could not properly parse the short
versions of the '--ds9geometry', '--ds9extra', '--ds9center' and
'--ds9mode' when the value was touching the short option name. Found
and fixed by Giacomo Lorenzetti.
- bug #65822: MakeProfiles does not write negative values with --replace.
- bug #65833: Convolve crashes when the kernel only has a single element.
- bug #65847: make check FAILs on color-faint-gray script when libjpeg
not found. Reported by Takashi Ichikawa.
- bug #65853: Statistics crashes with the --ontile option.
- bug #65862: Output directory ignored when Statistics is called with
the '--ontile' option.
- bug #65935: Arithmetic crashes with constants like speed of light.
- bug #65939: Column numbers in plain-text table metadata with +100
columns incorrectly written for the first 9 (single-digit) columns;
found with the help of Sepideh Eskandarlou.
- bug #65941: Name of spectral lines with wavelength were not rounded to
smallest nearest integer (see description of change in
CosmicCalculator).
* Noteworthy changes in release 0.22 (library 20.0.0) (2024-02-03)
** New publications
- https://ui.adsabs.harvard.edu/abs/2023RNAAS...7..269E by Eskandarlou et
al. on Gnuastro's zero point measurement script for calibration of
astronomical images ('astscript-zeropoint').
- https://ui.adsabs.harvard.edu/abs/2024RNAAS...8...10I by Infante-Sainz
& Akhlaghi on Gnuastro's script to visualize the full dynamic range of
astronomical images ('astscript-color-faint-gray').
- https://ui.adsabs.harvard.edu/abs/2024RNAAS...8...22I by Infante-Sainz
et al. on Gnuastro's script to measure the radial profile of a given
source ('astscript-radial-profile').
** New features
*** New program
- 'astscript-color-faint-gray': a new installed script to maximize the
visualization of the contents within your astronomical images. It does
this by showing the brighter parts of the image as color, intermediate
regions as black and the noisy/faint regions as gray/white.
*** All programs
- The following options are now available in all programs to allow
customization of which metadata is printed in the 0-th HDU of the FITS
output of all Gnuastro programs. Also see the "Changed features" list
for an important change in this regard to all Gnuastro output FITS
files.
--outfitsnoconfig: do not print any metadata in the 0-th HDU (including
the used options and their values, the date, the versions and git
commit of running directory).
--outfitsnodate: do not write the 'DATE' keyword.
--outfitsnoversions: do not write the versions of Gnuastro and
mandatory dependencies.
--outfitsnocommit: do not write the Git commit.
*** Arithmetic
--arguments: takes the name of plain-text file that contains the list of
arguments to the program. This option is critical if you have very long
(thousands) of operands and operators (which happen in large pipelines
where the arguments are constructed automatically). Without this
option, the shell is going to abort with an "Argument list too long"
error message.
- New operators:
- rotate-coord: given a 2D point's coordinates, return the coordinates
after it has been rotated by your requested angle around your
requested center (see documentation for example).
- mad: Median Absolute Deviation (MAD) stacking.
- madclip-mad: MAD after MAD-clipping stacking.
- madclip-std: Standard deviation after MAD-clipping stacking.
- madclip-mean: Mean after MAD-clipping stacking.
- madclip-median: Median after MAD-clipping stacking.
- madclip-number: Number of elements after MAD-clipping stacking.
- madclip-fill-mad: MAD after filled MAD-clipping stacking: this
involves a two-phase clipping: after the first
clipping, Arithmetic will "fill" the unclipped
holes in each input and masks them for a second
round. This is critical for clipping more
diffuse outliers: where a pixel may not be
clipped individually and will produce biased
results, but due to proximity with many nearby
clipped pixels, it can be discarded and thus its
effect on the final stack be removed.
- madclip-fill-std: Standard deviation after filled MAD-clipping stacking.
- madclip-fill-mean: Mean after filled MAD-clipping stacking.
- madclip-fill-median: Median after filled MAD-clipping stacking.
- madclip-fill-number: Num. of elements after filled MAD-clipping stacking.
- sigclip-fill-mad: MAD after filled MAD-clipping stacking.
- sigclip-fill-std: Standard deviation after filled MAD-clipping stacking.
- sigclip-fill-mean: Mean after filled MAD-clipping stacking.
- sigclip-fill-median: Median after filled MAD-clipping stacking.
- sigclip-fill-number: Num. of elements after filled MAD-clipping stacking.
- sigclip-mad: MAD after sigma-clipping stacking.
- collapse-madclip-mad: Collapse dim. by MAD-clipped MAD.
- collapse-madclip-std: Collapse dim. by MAD-clipped STD.
- collapse-madclip-mean: Collapse dim. by MAD-clipped mean.
- collapse-madclip-median: Collapse dim. by MAD-clipped median.
- collapse-madclip-number: Collapse dim. by MAD-clipped number.
- collapse-madclip-fill-mad: Collapse dim. by filled MAD-clipped MAD.
- collapse-madclip-fill-std: Collapse dim. by filled MAD-clipped STD.
- collapse-madclip-fill-mean: Collapse dim. by filled MAD-clipped mean.
- collapse-madclip-fill-median: Collapse dim. by filled MAD-clipped median.
- collapse-madclip-fill-number: Collapse dim. by filled MAD-clipped num.
- collapse-sigclip-mad: Collapse dim. by sigma-clipped MAD.
- collapse-sigclip-fill-mad: Collapse dim. by filled sigma-clipped MAD.
- collapse-sigclip-fill-std: Collapse dim. by filled sigma-clipped STD.
- collapse-sigclip-fill-mean: Collapse dim. by filled sigma-clipped mean.
- collapse-sigclip-fill-median: Collapse dim. by filled sigma-clip. median.
- collapse-sigclip-fill-number: Collapse dim. by filled sigma-clipped num.
*** Fits
--arguments: takes the name of plain-text file that contains the list of
arguments to the program. Currently this only works with '--keyvalue'
and is necessary when you want to get the keywords of a very long list
of files (~thousands!). Without this option, the shell is going to
abort with an "Argument list too long" error message.
--datasum-encoded: new option that will return an ASCII encoded
16-character string for the value of the 'DATASUM' concept in FITS data
verification.
*** Statistics
--checkskynointerp: similar to '--checksky', but stop as soon as the good
tiles are found (do not continue to interpolation and smoothing of the
tiles which can be time consuming if not needed).
--mad: Median Absolute Deviation (MAD) of input dataset.
--madclip: MAD-clipping, showing intermediate results.
--sigclip-mad: MAD of input after sigma-clipping.
--madclip-number: Number of input elements after MAD-clipping.
--madclip-median: Median of input elements after MAD-clipping.
--madclip-mean: Mean of input elements after MAD-clipping.
--madclip-std: Standard deviation of input elements after MAD-clipping.
--madclip-mad: MAD of input elements after MAD-clipping.
--mclipparams: Parameters of MAD-clipping, similar to sigma-clipping. Note
that in a Gaussian distribution, MAD = 0.67 sigma, so if
you want something similar to 3-sigma (the default), you
should set the MAD multiple to 5 (the default).
*** Makefile extension
- 'ast-text-to-upper': convert input string to upper-case.
- 'ast-text-to-lower': convert input string to lower-case.
*** Library
**** Functions
- gal_dimension_collapse_mclip_mad: MAD-clipped MAD.
- gal_dimension_collapse_mclip_fill_mad: filled MAD-clipped MAD.
- gal_dimension_collapse_mclip_std: MAD-clipped STD.
- gal_dimension_collapse_mclip_fill_std: filled MAD-clipped STD.
- gal_dimension_collapse_mclip_mean: MAD-clipped mean.
- gal_dimension_collapse_mclip_fill_mean: filled MAD-clipped mean
- gal_dimension_collapse_mclip_median: MAD-clipped median.
- gal_dimension_collapse_mclip_fill_median: filled MAD-clipped median.
- gal_dimension_collapse_mclip_number: MAD-clipped number.
- gal_dimension_collapse_mclip_fill_number: filled MAD-clipped number.
- gal_dimension_collapse_sclip_mad: sigma-clipped MAD
- gal_dimension_collapse_sclip_fill_mad: filled sigma-clipped MAD.
- gal_dimension_collapse_sclip_fill_std: filled sigma-clipped STD.
- gal_dimension_collapse_sclip_fill_mean: filled sigma-clipped mean
- gal_dimension_collapse_sclip_fill_median: filled sigma-clipped median.
- gal_dimension_collapse_sclip_fill_number: filled sigma-clipped number.
- gal_fits_hdu_datasum_encoded: the 16-character encoded sting datasum.
- gal_fits_key_list_add_date: add 'DATE' to current list of keywords.
- gal_fits_key_list_add_git_commit: add the Git commit information of the
running directory to the input list of keywords (if built with libgit2).
- gal_fits_key_list_add_software_versions: add versions of Gnustro and
its mandatory dependencies to the input list of keywords.
- gal_statistics_clip_mad: MAD clipping of given input.
- gal_statistics_mad: return median absolute deviation (MAD).
- gal_statistics_median_mad: return median and MAD.
- gal_txt_read_to_list: read all the space-separated words of the input
plain-text file as a separate node in a linked list of strings.
**** Macros
- Used by 'gal_arithmetic':
- GAL_ARITHMETIC_OP_SIGCLIP_MAD: Sigma-clipped MAD.
- GAL_ARITHMETIC_OP_SIGCLIP_MAD: Sigma-clipped STD.
- GAL_ARITHMETIC_OP_SIGCLIP_FILL_NUMBER: MAD-clipped num. of arrays.
- GAL_ARITHMETIC_OP_SIGCLIP_FILL_MEAN: MAD-clipped mean of arrays.
- GAL_ARITHMETIC_OP_SIGCLIP_FILL_MEDIAN: MAD-clipped median of arrays.
- GAL_ARITHMETIC_OP_SIGCLIP_FILL_STD: MAD-clipped STD of arrays.
- GAL_ARITHMETIC_OP_SIGCLIP_FILL_MAD: MAD-clipped STD of arrays.
- GAL_ARITHMETIC_OP_MADCLIP_NUMBER: MAD-clipped number of mult. arrays.
- GAL_ARITHMETIC_OP_MADCLIP_MEAN: MAD-clipped mean of multiple arrays.
- GAL_ARITHMETIC_OP_MADCLIP_MEDIAN: MAD-clipped median of mult. arrays.
- GAL_ARITHMETIC_OP_MADCLIP_STD: MAD-clipped STD of multiple arrays.
- GAL_ARITHMETIC_OP_MADCLIP_MAD: MAD-clipped STD of multiple arrays.
- GAL_ARITHMETIC_OP_MADCLIP_FILL_NUMBER: MAD-clipped num. of arrays.
- GAL_ARITHMETIC_OP_MADCLIP_FILL_MEAN: MAD-clipped mean of arrays.
- GAL_ARITHMETIC_OP_MADCLIP_FILL_MEDIAN: MAD-clipped median of arrays.
- GAL_ARITHMETIC_OP_MADCLIP_FILL_STD: MAD-clipped STD of arrays.
- GAL_ARITHMETIC_OP_MADCLIP_FILL_MAD: MAD-clipped STD of arrays.
- Used by 'gal_statistics_clip_sigma' and 'gal_statistics_clip_mad':
- GAL_STATISTICS_CLIP_OUTCOL_NUMBER_USED: index of final number in output.
- GAL_STATISTICS_CLIP_OUTCOL_MEAN: index of clipped mean in output.
- GAL_STATISTICS_CLIP_OUTCOL_STD: index of clipped STD in output.
- GAL_STATISTICS_CLIP_OUTCOL_MEDIAN: index of clipped median in output.
- GAL_STATISTICS_CLIP_OUTCOL_MAD: index of clipped MAD in output.
- GAL_STATISTICS_CLIP_OUTCOL_NUMBER_CLIPS: index of clipped number in output.
- GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MEAN: bit flag for measuring mean.
- GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_STD: bit flag for measuring STD.
- GAL_STATISTICS_CLIP_OUTCOL_OPTIONAL_MAD: bit flag for measuring MAD.
** Removed features
*** Library
- gal_fits_key_write_version: redundant with the new
'gal_fits_key_list_add_software_versions'.
- gal_fits_key_write_version_in_ptr: as in gal_fits_key_write_version.
- gal_fits_key_write_config: redundant since all configuration (options)
are stored as generic keys.
** Changed features
*** All programs
- Date and versions FITS keywords are only written in the 0-th HDU of
output FITS files (along with the options names and values), not the
HDU(s) containing data. Until now, the data and versions metadata were
written in the data HDU. However, ultimately, these are also metadata
and are better suited for the 0-th HDU. For example when everything
else is identical, you expect an exactly reproducible HDU (to verify
with the FITS 'CHECKSUM' for example), but every time you run, 'DATE'
will be different! Similarly for the versions of the software or Git
commit (they may change, while your dataset doesn't). To disable these
in the 0-th HDU also, see the newly added '--outfitsno*' options in the
new features.
*** Arithmetic
- Binary operators (like '+' or 'x') that are given two integers will
crash with an error if the input operands have the same width but
different signs. Until now, they would just report a warning and print
the output. However, in large scripts, users could miss the warning and
not be aware of a possibly wrong result. Therefore it is more robust to
crash with an error rather than print a warning. This was suggested by
Sepideh Eskandarlou and implemented by Faezeh Bidjarchian after a poll
on Gnuastro's Matrix chat channel (#gnuastro:openastronomy.org).
*** CosmicCalculator
- In a non-FLRW model (when the sum of the densities is not 1.0),
CosmicCalculator no longer crashes with an error since calculations
like the radial comoving distance ('--properdistance') are still valid
in a curved cosmology. Instead a warning is printed informing the user
that with Gnuastro's current implementation angular diameter based
calculations will be wrong. This was implemented by Boud Roukema.
--angulardiamdist is the new name for the old '--angulardimdist'
option. This was necessary to avoid confusion of 'dim' in the old name
with "dimension"; 'diam' makes it clear that this is a "diameter", not
'dimension'. This was suggested by Boud Roukema.
*** Crop
- The 'ICF*' keywords are now written in the 0-th HDU of Crop's outputs
(the extension with no data, only metadata). Therefore, the WCS is now
the only metadata in the HDU containing data.
*** astscript-psf-stamp
- When no name is given to the temporary directory, the default name will
contain the center coordinates as a suffix (to avoid being confused
with other calls for other centers), and the contents of the temporary
directy will have the same filename. Until now, the temporary directory
name would be the same for different calls, and the coordinates suffix
would be in the temporary file names. This could cause confusions and
bugs (for example one call to this script could delete the temporary
directory that was being used by a later call).
*** Library
- gal_cosmology_age: a 'quiet' argument is added to disable warnings.
- gal_cosmology_proper_distance: similar 'gal_cosmology_age'.
- gal_cosmology_comoving_volume: similar 'gal_cosmology_age'.
- gal_cosmology_critical_density: similar 'gal_cosmology_age'.
- gal_cosmology_angular_distance: similar 'gal_cosmology_age'.
- gal_cosmology_luminosity_distance: similar 'gal_cosmology_age'.
- gal_cosmology_distance_modulus: similar 'gal_cosmology_age'.
- gal_cosmology_to_absolute_mag: similar 'gal_cosmology_age'.
- gal_fits_key_write: to generalize, the "title" argument has been
removed (because it is a keyword). It can also create the file if it
doesn't exist and can optionally free the list of keywords (until now,
it would always free them).
- gal_fits_key_write_in_ptr: similar to 'gal_fits_key_write'.
- gal_fits_img_write: no more 'program_name'; it is a keyword title;
added 'freekeys' argument to optionally free the input keyword list.
- gal_fits_img_write_to_type: as in 'gal_fits_img_write'.
- gal_fits_img_write_corr_wcs_str: as in 'gal_fits_img_write'.
- gal_fits_tab_write: as in 'gal_fits_img_write'.
- GAL_STATISTICS_CLIP_MAX_CONVERGE: new name for the old
'GAL_STATISTICS_SIG_CLIP_MAX_CONVERGE'. Since we now also have
MAD-clipping.
- gal_statistics_clip_sigma: new name for 'gal_statistics_sigma_clip'.
- gal_table_write: as in 'gal_fits_img_write'.
- gal_tile_full_values_write: as in 'gal_fits_img_write'.
- gal_wcs_write: as in 'gal_fits_img_write'.
** Bugs fixed
- bug #46225: Programs don't depend on libs in Makefiles; reported by
Mosè Giordano and fixed by Thorsten Alteholz.
- bug #52295: Cosmology library integrals crash for high z; fixed by
Thorsten Alteholz.
- bug #52674: Correcting Clang warnings on macOS during compilation;
fixed with help of Thorsten Alteholz and Raul
Infante-Sainz.
- bug #61476: astscript-ds9-region: output created when input file does
not exist; reported by Sepideh Eskandarlou, fixed by
Thorsten Alteholz.
- bug #64825: astscript-fits-view only takes the last HDU provided;
reported by Teet Kuumta and fixed by Raul Infante-Sainz.
- bug #64852: astscript-zeropoint: crash when input is in 0-th HDU;
reported in https://savannah.gnu.org/support/?110952; fixed
by Sepideh Eskandarlou.
- bug #64915: Segment's SKY_STD output HDU does not have WCS; reported by
Sepideh Eskandarlou.
- bug #64952: Table error message for incorrect column name was not
complete; reported by Sepideh Eskandarlou.
- bug #65005: astscript-psf-unit not accounting for --innerhdu; reported
by Ignacio Ruiz Cejudo.
- bug #65050: Use --libtool option in the TEMPLATE usage; found and fixed
by Faezeh Bidjarchian.
- bug #65055: Columns of empty FITS tables (with no rows) read in
reverse.
- bug #65084: Fits program's --skycoverage prints connected numbers for
cubes; found with help of Rahna Payyasseri Thanduparackal.
- bug #65106: Necessity of --oversample with astscript-radial-profile
when a 2D profile is to be created with '--customtable' of
MakeProfile; reported by Sepideh Eskandarlou.
- bug #65141: MakeCatalog crash when labled image is all blank; reported
by Sepideh Eskandarlou.
- bug #65149: MakeCatalog '--frac-max*-*' measurements done without
'--frac-max'; reported by Helena DomÃnguez Sánchez.
- bug #65194: --quiet does not disable the warning for un-given
minmapsize; reported by Boud Roukema.
- bug #65195: CosmicCalculator prevents non-flat FLRW models and angular
diameter distance only correct in flat models. Found and
fixed by Boud Roukema.
* Noteworthy changes in release 0.21 (library 19.0.0) (2023-10-20)
** New features
*** New program
- 'astscript-pointing-simulate': a new installed script that will
simplify the process of designing a pointing pattern for your observing
strategy. It takes into account the distortion of the camera, runs in
parallel and allows customizations (for example to account for
vignetting, or large blobs of bad pixels) through hooks at relevant
steps. A new tutorial has been added in the Tutorials chapter of the
book on how to use this new script. The following research note has
also been published for this script:
https://ui.adsabs.harvard.edu/abs/2023RNAAS...7..211A
*** Book
- New tutorial on a low surface brightness optimized design of a pointing
pattern using the newly added installed script in Gnuastro for
simulating the exposure map of a pointing pattern stack (it is called as
'astscript-pointing-simulate' on the command-line).
- Smaller tutorials that were distributed within the documentation of
different programs are brought into the "Tutorials" chapter.
- New "Standard deviation vs. error" sub-section added under the
MakeCatalog section. It uses real examples to clearly show the
fundamental difference between the two (which are sometimes confused
with each other). This was written with the help of Raul Infante-Sainz.
*** Configuration files
- To separate the option name and value, you can now also use the '='
character. This allows your custom configuration files to also be
loaded into a shell (with 'source') or Make (with 'include'). Until
now, only white-space characters were allowed as separators of option
names and values.
--config-prefix=STR: Given string can be a prefix of options in
configuration files. Added with the point above, this allows loading
the different configuration files of different instances of the same
program without overwriting them. See the example in the book.
*** Arithmetic
- New operators (see book for a full description):
- pool-min: Min-pooling to reduce the size of the input by calculating
the minimum of a the pixels within the pooling window
(accounting for a stride). See the new "Pooling operators"
section of the book for more. The pooling operators were
all implemented by Faezeh Bidjarchian.
- pool-max: Similar to 'pool-min' but using maximum.
- pool-sum: Similar to 'pool-min' but using sum.
- pool-mean: Similar to 'pool-min' but using mean.
- pool-median: Similar to 'pool-min' but using median.
- to-1d: convert the input operand into a 1D array, no matter how many
dimensions it has. Suggested by Faezeh Bidjarchian.
- trim: remove all fully-blank outer regions of the input dataset.
- eq-b1950-to-eq-j2000: Convert input equatorial coordinates (RA and
Dec) in the B1950 equinox to equatorial J2000
equinox.
- eq-b1950-to-ec-b1950: Same input as above; to ecliptic B1950.
- eq-b1950-to-ec-j2000: Same input as above; to ecliptic J2000.
- eq-b1950-to-galactic: Same input as above; to Galactic.
- eq-b1950-to-supergalactic: Same input as above; to Supergalactic.
- eq-j2000-to-eq-b1950: Convert input equatorial coordinates (RA and
Dec) in the J2000 equinox to equatorial B1950
equinox.
- eq-j2000-to-ec-b1950: Same input as above; to ecliptic B1950.
- eq-j2000-to-ec-j2000: Same input as above; to ecliptic J2000.
- eq-j2000-to-galactic: Same input as above; to Galactic.
- eq-j2000-to-supergalactic: Same input as above; to Supergalactic.
- ec-b1950-to-eq-b1950: Convert input ecliptic coordinates in the
B1950 equinox to equatorial B1950 equinox.
- ec-b1950-to-eq-j2000: Same input as above; to equatorial B1950.
- ec-b1950-to-ec-j2000: Same input as above; to ecliptic J2000.
- ec-b1950-to-galactic: Same input as above; to Galactic.
- ec-b1950-to-supergalactic: Same input as above; to Supergalactic
- ec-j2000-to-eq-b1950: Convert input ecliptic coordinates in the
J2000 equinox to equatorial B1950 equinox.
- ec-j2000-to-eq-j2000: Same input as above; to equatorial J2000.
- ec-j2000-to-ec-b1950: Same input as above; to ecliptic B1950.
- ec-j2000-to-galactic: Same input as above; to Galactic.
- ec-j2000-to-supergalactic: Same input as above; to Supergalactic.
- galactic-to-eq-b1950: Convert input Galactic coordinates to
equatorial B1950 equinox.
- galactic-to-eq-j2000: Same input as above; to equatorial J2000.
- galactic-to-ec-b1950: Same input as above; to ecliptic B1950.
- galactic-to-ec-j2000: Same input as above; to ecliptic J2000.
- galactic-to-supergalactic: Same input as above; to Supergalactic.
- supergalactic-to-eq-b1950: Convert input Supergalactic coordinates to
equatorial B1950 equinox.
- supergalactic-to-eq-j2000: Same input as above; to equatorial J2000.
- supergalactic-to-ec-b1950: Same input as above; to ecliptic B1950.
- supergalactic-to-ec-j2000: Same input as above; to ecliptic J2000.
- supergalactic-to-galactic: Same input as above; to Galactic.
*** ConvertType
- It is now possible to write TIFF files in the output (until now,
Gnuastro could only read TIFF files). This step was written by Fathma
Mehnoor.
*** Convolve
--conv-on-blank: do not ignore blank pixels in the convolution. This will
effectively expand the non-blank regions of your dataset into the blank
regions and only works in spatial-domain convolution. This was
suggested by Raul Infante-Sainz.
*** MakeCatalog
- New measurements:
--river-min: minimum river value around a clump.
--river-max: minimum river value around a clump.
*** Table
--info-num-cols: print the number of the input table's columns and abort.
--info-num-rows: print the number of the input table's rows and abort.
- '$_all' in column arithmetic: when an arithmetic expression contains
this string, it will be repeated independently for all the columns of
the input table.
- New operators in Table's column arithmetic:
- eq-j2000-to-flat: convert the RA and Dec columns in a table to a
flat-RA and flat-Dec (accounting for a reference point as well as any
type of projection that is available in the FITS WCS standard). This
is necessary when you are plotting points in a report or paper cover
more than a couple of square degrees on the sky. Without this
operator, significant distortions will appear when you plot spherical
coordinates (RA,Dec) on a flat plane (your paper's plot). See the
documentation for this operator in the book for more.
- eq-j2000-from-flat: the inverse of 'eq-j2000-to-flat' (useful when
designin dither patterns for example (where you know the final
shape/distanced after flatting; and want coordinates on the sphere).
*** astscript-zeropoint
--mksrc: use a custom Makefile for estimating the zeropoint, not the
default installed Makefile. This is primarily intended for debugging or
developing this script, not for normal usage.
*** Library
- gal_blank_not_minmax_coords: returns the minimum/maximum coordinates of
non-blank regions of input dataset.
- gal_blank_trim: trim all outer blank regions from the input dataset.
- gal_polygon_area_sky: area of polygon defined on celestial coordinates.
- gal_pool_min: min-pooling function, see 'pool-min' above.
- gal_pool_max: max-pooling function, see 'pool-min' above.
- gal_pool_sum: sum-pooling function, see 'pool-min' above.
- gal_pool_mean: mean-pooling function, see 'pool-min' above.
- gal_pool_median: median-pooling function, see 'pool-min' above.
- gal_statistics_mad: Calculate the Median Absolute Deviation (MAD).
- gal_statistics_median_mad: Calculate the Median and MAD.
- gal_statistics_clip_mad: Median Absolute Deviation (MAD) clipping.
- gal_tiff_write: write 'gal_data_t' to a TIFF file; written by
Fathma Mehnoor.
- gal_wcs_projection_name_to_id: convert projection names to IDs.
- gal_wcs_projection_name_from_id: convert projection IDs to their name.
- gal_wcs_coordsys_convert_points: convert the input set of points from
one celestial coordinate system to another.
- gal_wcs_coordsys_sys1_ref_in_sys2: return the longitude reference point
of the first celestial coordinate system in the second coordinate
system's longitude and latitude.
** Removed features
*** MakeNoise
- MakeNoise (with executable name 'astmknoise') has been removed from
Gnuastro. This is because the 'mknoise-*' operators of Arithmetic do
the same job, but in a much more customizable, modular and manageable
manner; allowing to mix with other operators. The "Noise basics"
sub-section of the book has been moved under the Arithmetic program's
documentation also.
** Changed features
*** Arithmetic
- mknoise-sigma-from-mean: new name for the old 'mknoise-poisson'
random number distribution.
- mknoise-poisson: now produces correct Poisson distribution output,
which can be non-symmetric in low background values and is always
integers. For a complete description and hands-on example of the
differences with 'mknoise-sigma-from-mean' and 'mknoise-sigma', see
the description of these three operators in the "Random number
generators" operators sub-section of the "Arithmetic operators"
section of the Gnuastro book.
*** MakeCatalog
- The dash in the column names of the following measurement names has
been replaced by underscore to conform with the general standard of
using underscores between words in column names (reported by Raul
Infante-Sainz). Also, in the case of '--sigclip-mean-sb-delta' the
column name didn't correspond to the option name.
--------------------------------------------------------------
Option Column name Old column name
--------------------------------------------------------------
--sigclip-mean SIGCLIP_MEAN SIGCLIP-MEAN
--sigclip-median SIGCLIP_MEDIAN SIGCLIP-MEDIAN
--sigclip-number SIGCLIP_NUMBER SIGCLIP-NUMBER
--sigclip-std SIGCLIP_STD SIGCLIP-STD
--sigclip-mean-sb SIGCLIP_MEAN_SB SIGCLIP-MEAN-SB
--sigclip-mean-sb-delta SIGCLIP_MEAN_SB_DELTA SIGCLIP-MEAN-SB-ERR
--------------------------------------------------------------
- The default extension name for the values image ('--valueshdu') is now
'INPUT-NO-SKY'. Until now, it was simply '1' (without any hint on what
what is expected by default). Recall that if the sky is not subtracted
from the values image you can use '--sky' for MakeCatalog to subtract
it before the measurements. Implemented by Sepideh Eskandarlou.
- When the input clump labels do not have the 'NUMLABS' keyword
(specifying how many clumps there are in the image, independent of the
number of objects), MakeCatalog will no longer re-label the input
clumps: the output catalog will have the same clump ids as the
labels. Until now, it would generate new labels for the clumps in such
cases (along with a new FITS image containing the clumps with the new
labels to allow you to match them). The output catalog also had the new
label, not the original labels (the old behavior was therefore very
inconvenient!).
*** Segment
- The default name of the first extension of Segment's output is now
'INPUT-NO-SKY', until now, it was 'INPUT'. This was necessary because
the first extension of the output is not the exact input file (if a
value is given to '--sky' it will be subtracted from the input in the
first extension). Implemented by Sepideh Eskandarlou.
*** astscript-psf-stamp
- After applying the signal-to-noise threshold (if it is requested), any
extra pixels that are not connected to the central target are also
masked. Such pixels can remain in rivers between bright clumps and
would cause problem in the final stack and were not removed until now.
Implemented by Sepideh Eskandarlou
*** Library
- gal_convolve_spatial: new argument to allow convolution over blank
elements.
- gal_convolve_spatial_correct_ch_edge: new argument to allow convolution
over blank elements.
- gal_polygon_area_flat: new name for the old 'gal_polygon_area'
function. This was necessary because this function assumes flat
coordinates and can lead to wrong results when used with a polygon
defined by RA,Dec for example.
- gal_statistics_clip_sigma: new name for 'gal_statistics_sigma_clip',
also, a new argument has been added to measure statistics that were not
automatically calculated during the clipping.
- gal_wcs_distortion_name_to_id: new name of gal_wcs_distortion_from_string.
- gal_wcs_distortion_name_from_id: new name of gal_wcs_distortion_to_string.
- gal_wcs_coordsys_name_to_id: new name of gal_wcs_coordsys_from_string.
** Bugs fixed
bug #64138: Arithmetic's mknoise-poisson only using first pixel value.
Reported by Irene Pintos Castro.
bug #64146: SIGCLIP column names contain dashes instead of
underlines. Reported by Raul Infante-Sainz.
bug #64148: MakeCatalog crash when using the options '--std' and/or
'sigclip-mean-sb-delta' for clumps catalog. Reported by Raul
Infante-Sainz.
bug #64153: astscript-ds9-region: '--namecol' gives a constant zero in
image-mode. Reported by Zahra Sharbaf.
bug #64186: Query: core dump when custom name given to '--dataset'. First
reported by Zohreh Ghaffari as a problem in
'astscript-select-stars' (sr #110874) and identified as a
problem in Query by Sepideh Eskandarlou.
bug #64199: info astscript-zeropoint not working. Reported by Zahra
Sharbaf.
bug #64246: Query's --information gives a crash for Gaia datasets.
Reported by Rashid Yaaqib.
bug #64250: astscript-fits-view: no message printed when input file did
not exist. Reported by Ryan Begley.
bug #64274: MakeProfiles crash for points in 3D. Found by Teet Kuumta.
bug #64303: Statistics fit --fitestimate crash when covariance matrix has
NaN. Reported Sepideh Eskandarlou.
bug #64357: Plain text metadata ignored when no name is present.
bug #64392: Arithmetic's isnotblank operator producing zeros when no
blanks present.
bug #64420: Error message of wrongly specified HDU doesn't give the
problematic option name; reported by Aaron Watkins.
bug #64431: gal_wcs_to_cd does not account for altlin==3. Reported by
Colin Orion Chandler.
bug #64541: Segment's --keepmaxnearriver has no effect.
bug #64544: Arithmetic crashes with tofilefree operator value has a
directory.
bug #64545: Accounting for CFITSIO silently appending '.gz' when file
doesn't exist. Reported by Sepideh Eskandarlou.
bug #64610: --copykeys of Fits program finds wrong key when the first
characters of the keyword names match.
bug #64615: Fits program's '--pixelareaonwcs' only accurate on equator.
bug #64635: astscript-psf-stamp's '--snthresh' option produces fully NaN
valued image when a saturated star is in the center. Found
and fixed by Sepideh Eskandarlou.
bug #64643: MakeCatalog not reporting the actual '--upmaskhdu' that was
used on standard output. Found by S. Zahra Hosseini
Shahisavandi.
* Noteworthy changes in release 0.20 (library 18.0.0) (2023-04-29)
** New features
*** New program
- astscript-zeropoint: this script is used for calibrating the pixel
values in one image (finding its "zero point") based on a reference
catalog or (any number of) reference image(s). The script allows for
checking many apertures in the same command to let you choose the
best; and will operate in parallel where necessary. The documentation
of this script contains two tutorials on how to optimally use it and
interpret its results; please read and run them once before using it.
This script was written as a team effort lead by Sepideh Eskandarlou
and including (in alphabetical order): Elham Saremi, Giulia Golini,
Raul Infante-Sainz, Samane Raji, Zahra Sharbaf, Zohreh Ghaffari.
*** Book
- New tutorial on detection and spectrum extraction from 3D data has
been added in the "Tutorials" chapter. It uses a MUSE cube and
demonstrates how to subtract the continuum, run NoiseChisel, Segment
and MakeCatalog for obtaining spectra and creating pseudo narrow-band
images of emission-lines.
- New "Shell tips" sub-section added (under the "Command-line" section
of the "Common program behavior" chapter). This sub-section contains
useful shell tips and tricks that can be useful when using Gnuastro's
programs.
*** Arithmetic
--writeall: Write all datasets on the stack as separate HDUs in the
output; this is useful in debugging incomplete Arithmetic commands.
- New operators (also available in Table).
- isnotblank: same as 'isblank not', but slightly more efficient.
This was suggested by Sepideh Eskandarlou.
- swap: swap the top two datasets on the stack of operands.
- index: return dataset of same size, with pixel values that are
integers starting from 0 and incrementing one by one. This operator
doesn't pop the top dataset from the stack, it only adds one (to
avoid extra RAM usage in most usage scenarios).
- counter: similar to 'index', but integers start with 1.
- indexonly: similar to 'index', but pops the top stack dataset.
- counteronly: similar to 'counter', but pops the top stack dataset.
- constant: replace all elements of input image or column with the
given constant value (for example to set all pixels in an image to
NaN or zero), or to give identifiers to each source catalog when
adding rows from different catalogs, see the documentation for
complete examples.
- counts-to-nanomaggy: convert counts to Nanomaggy (with fixed
zeropoint of 22.5, used as the pixel units of many surveys like
SDSS). This was suggested by Giulia Golini.
- nanomaggy-to-counts: convert nanomaggy to counts. This was suggested
by Giulia Golini.
- box-vertices-on-sphere: calculate the RA,Dec of vertices of the four
vertices of a rectangle from its center and width along RA and
Dec. This takes into account the curved nature of the coordinate
system. Added after discussion with Martin Kuemmel.
- number-neighbors: Return the number of non-zero neighbors of each
non-zero pixel in a binary image.
- New operators (only in the Arithmetic program):
- interpolate-meanngb: interpolate blank values with mean of the
requested number of nearest neighbors.
- Alternative (shorter) names for existing operators, added after
discussion with Samane Raji.
- u8: same as 'uint8' (to convert to unsigned 8-bit integers).
- i8: same as 'int8' (to convert to signed 8-bit integers).
- u16: same as 'uint16' (to convert to unsigned 16-bit integers).
- i16: same as 'int16' (to convert to signed 16-bit integers).
- u32: same as 'uint32' (to convert to unsigned 32-bit integers).
- i32: same as 'int32' (to convert to signed 32-bit integers).
- u64: same as 'uint64' (to convert to unsigned 64-bit integers).
- i64: same as 'int64' (to convert to signed 64-bit integers).
- f32: same as 'float32' (to convert to 32-bit floating point).
- f64: same as 'float64' (to convert to 64-bit floating point).
*** CosmicCalculator
- Spectral line database has been increased to 235 spectral lines in the
UV and optical (previously only had 41 lines!) from:
http://astronomy.nmsu.edu/drewski/tableofemissionlines.html
To see the full list, run 'astcosmiccal --listlines'.
--lineunit: allows you to specify the units of the displayed spectral
line wavelengths. Currently, it take four units: 'm', 'micron', 'nm'
and 'angstrom'. We can add any other unit easily, just let us know.
*** Crop
--append: if the output file already exists, append the cropped image
HDU to the already existing HDUs of the file. Without this option, any
existing HDUs in the output file will be removed (default behavior).
--metaname: Specify the name of the cropped output HDU (value to the
'EXTNAME' keyword in FITS).
*** MakeCatalog
- Book: with the increasing number of possible measurements the
"MakeCatalog measurements" section of the Gnuastro book has been
broken into separate sub-section and contextually similar measurements
are grouped separately there.
--sigclip-mean-sb: surface brightness (over one pixel's area in
arcsec^2) of the sigma-clipped mean of the values. This is useful in
scenarios where you want to compare surface brightness values
independent of the object's area, and by removing outliers (for
example in a radial profile). See the description of this option in
the book for more. This was added after a discussion by Ignacio
Trujillo and Zahra Sharbaf.
--sigclip-mean-sb-err: the error in '--sigclip-mean-sb' but using the
sigma-clipped standard deviation of the values as the error, not the
standard deviation image.
--sigclip-std-sb: the surface brightness (over one pixel's area in
arcsec^2) of the sigma-clipped standard deviation of the values. This
can be used to find the reliable surface brightness of a radial
profile for example.
--sum-in-slice: [3D in; vector out] Sum of values in each slice.
--sum-err-in-slice: [3D in; vector out] Error in '--suminslice'.
--area-in-slice: [3D input; vector out] Number of labeled in each slice.
--sum-proj-in-slice: [3D input; vector out] Sum of projected area
in each slice.
--area-proj-in-slice: [3D in; vector out] Number of voxels that are used
in '--sum-proj-in-slice'.
--sum-proj-err-in-slice: [3D in; vector out] Error of '--sum-proj-in-slice'.
--area-other-in-slice: [3D in; vector out] Area of other label in
projected area on each slice.
--sum-other-in-slice: [3D in; vector out] Sum of other label in
projected area on each slice.
--sum-other-err-in-slice: [3D in; vector out] Area in
'--sum-other-in-slice'.
*** NoiseChisel
--outliernumngb: the number of neighboring tiles to reject those that
have passed (the mean-median quantile difference criteria) because of
being on the wings of bright stars/galaxies. Until now, this number
was the same as the number given to '--interpnumngb', but it is
confusing and can cause problems (because interpolation is done after
outlier rejection). From this version, '--interpnumngb' is only in
charge of the number of neighboring tiles for interpolation outliers
are defined based on the number of tiles given to this
option. Therefore, if you are not detecting the wings of large
galaxies, THE BEST solution is most-probably to increase
'--outliernumngb'. This was done after a discussion with Elham Saremi.
*** Statistics
--outliernumngb: see description of same option in NoiseChisel.
*** Table
- Vector columns with multiple values per column are now supported. The
following features have been added to help working on vector columns:
- Book: a new "Vector columns" section has been added under the Table
program which describes the core concepts and usage examples of the
options below.
--information: also identifies vector columns (a column with more than
one value), by placing an '(N)' after the type. Where 'N' is the
number of elements within the vector.
--tovector: merge multiple single-valued columns into a new vector
column.
--fromvector: extract identified elements/tokens of a vector column to
separate single-valued columns.
--keepvectfin: do not delete the input columns to '--tovector' and
'--fromvector'.
--rowfirst: do row-based operations before column-based operations (by
default column-based operations are done first).
--transpose: column-based operator to transpose vector columns into a
new table that has the inverse number of rows and columns.
--txteasy: (or '-Y') when output is a plain-text file or just gets
printed on standard output (terminal), all floating point columns are
printed in fixed point notation (as in '123.456') instead of the
default exponential notation (as in '1.23456e+02'). For 32-bit
floating points, use a precision of 3 digits and for 64-bit floating
points use a precision of 6 digits. This can be useful for human
readability, but be careful with some scenarios (for example
'1.23e-120', which will show only as '0.0'!). For more, see the
changes in Table in this version.
- New column arithmetic operator:
- sorted-to-interval: return two columns from a single (sorted) input,
containing the minimum and maximum values of an interval. The
intervals are defined such that the intervals of every row fully
cover the full range of values in the given column. This can be
useful in scenarios where you need to build the inputs of the
'--customtable' feature in MakeProfiles (to build a 2D image a
custom profile).
*** astscript-radial-profile
--precision: sample the radial profile at precision less than one
pixel. This is useful when you need to sample the profile within the
central few pixels more accurately than integer pixels. See the
documentation of this option in the book for a complete explanation
with examples.
*** Library
- GAL_ARITHMETIC_OP_SWAP: swap the top two operands.
- GAL_ARITHMETIC_OP_INDEX: An index (counting from 0) for every element.
- GAL_ARITHMETIC_OP_INDEXONLY: Similar to 'GAL_ARITHMETIC_OP_INDEX'.
- GAL_ARITHMETIC_OP_COUNTER: A counter (counting from 1) for every element.
- GAL_ARITHMETIC_OP_COUNTERONLY: Similar to 'GAL_ARITHMETIC_OP_COUNTER'.
- GAL_ARITHMETIC_OP_COUNTS_TO_NANOMAGGY: convert counts to nanomaggy.
- GAL_ARITHMETIC_OP_NANOMAGGY_TO_COUNTS: convert nanomaggy to counts.
- GAL_ARITHMETIC_OP_BOX_VERTICES_ON_SPHERE: calculate the coordinates of
vertices of a rectangle on a sphere from its center and width/height.
- gal_binary_number_neighbors: num. non-zero neighbors of non-zero pixels.
- gal_blank_flag_not: binary dataset with 1 for those input pixels that
were blank.
- gal_data_alloc_empty: Allocate an empty dataset with a given number of
dimensions.
- gal_list_f64_to_data: convert list of float64s to a 'gal_data_t'
dataset with the requested type.
- gal_list_data_remove: Remove the given dataset from the given list.
- gal_list_data_select_by_id: find/select a dataset from a list of
datasets using an identification string (either counter or name).
- gal_permutation_apply_onlydim0: When we have a 2D input, apply
permutation for all the elements of each row (along dimension-0 in C).
- gal_statistics_has_negative: see if input has a negative value.
- gal_table_col_vector_extract: extract the given elements of a vector
column into separate columns.
- gal_table_cols_to_vector: merge multiple columns into a vector column.
- gal_units_counts_to_nanomaggy: Convert counts to nanomaggy.
- gal_units_nanomaggy_to_counts: Convert nanomaggy to counts.
- gal_wcs_box_vertices_from_center: calculate the coordinates of
vertices of a rectangle on a sphere from its center and width/height.
** Removed features
*** MakeCatalog
- Removed the short option format for the following measurements
(columns). This was done because the short options affect the sorting
in the output of '--help', making it hard to find the options
visually. Because most measurements do not have a short format, we have
seen that the short format is rarely used. Only the most commonly used
measurements keep their short format, and only when they don't affect
the alphabetic sorting of the output of '--help':
--area (short option was 'a')
--axis-ratio (short option was 'Q')
--magnitude-error (short option was 'G')
--num-clumps (short option was 'c')
--position-angle (short option was 'p')
--semi-major (short option was 'A')
--semi-minor (short option was 'B')
--sn (short option was 'n')
--sum (short option was 'b')
--upperlimit-mag (short option was 'u')
** Changed features
*** Configuration
--with-python: this has replaced the old '--without-python' option. The
Python extension features in the Gnuastro library are no longer built
by default. To have them, you need to explicitly configure Gnuastro
with this option. This was necessary because Python has many
peculiarities that were causing problems for users that don't need it
in Gnuastro. These features are only necessary for those who compile
pyGnuastro from source, most future users of pyGnuastro will be
installing pre-built binaries it through services like PyPI, so they
won't be needing it either.
*** CosmicCalculator
- Given the major increase of the spectral line database, the old line
names have been changed to accommodate the full database. For example
'H-alpha' has replaced 'halpha'. Please run 'astcosmiccal --listlines'
to see the new names for the previous lines.
*** MakeCatalog
- "Sum" used instead of "brightness"
--sum: new name for the old '--brightness' column. "Brightness" has a
specific meaning in astronomy/physics and has units of
energy/time. However, astronomical images aren't in this unit, they
are usually in counts, or can have an arbitrary units (like
nanomaggy). The new name is derived from what this column does (sum
of pixel values) to avoid confusions.
--sum-error: new name for '--brightnesserr'.
--clumps-sum: new name for '--clumpbrightness'.
--sum-no-river: new name for '--brightnessnoriver'.
--checkuplim: now returns the center of each label's tile, not the first
pixel (the bottom-left pixel position of the tile). The old behavior to
return the first pixel was un-intuitive, would require several manual
steps (to measure the minimum and maximum positions of each tile,
measure the width along each dimension and add it), and could lead to
confusing results when ignored. See the updated documentation of this
option for more on custom usage in special situations.
- Using a hyphen to help in readability and usability:
Old name New name
======== ========
--areaarcsec2 --area-arcsec2
--areamaxv --area-max-val
--areaminv --area-min-val
--areaxy --area-xy
--axisratio --axis-ratio
--clumpsarea --clumps-area
--clumpsgeow1 --clumps-geo-w1
--clumpsgeow2 --clumps-geo-w2
--clumpsgeow3 --clumps-geo-w3
--clumpsgeox --clumps-geo-x
--clumpsgeoy --clumps-geo-y
--clumpsgeoz --clumps-geo-z
--clumpsmagnitude --clumps-magnitude
--clumpsw1 --clumps-w1
--clumpsw2 --clumps-w2
--clumpsw3 --clumps-w3
--clumpsx --clumps-x
--clumpsy --clumps-y
--clumpsz --clumps-z
--fracmax --frac-max
--fracmaxarea1 --frac-max1-area
--fracmaxarea2 --frac-max2-area
--fracmaxradius1 --frac-max1-radius
--fracmaxradius2 --frac-max2-radius
--fracmaxsum1 --frac-max1-sum
--fracmaxsum2 --frac-max2-sum
--geoarea --geo-area
--geoareaxy --geo-area-xy
--geoaxisratio --geo-axis-ratio
--geopositionangle --geo-position-angle
--geosemimajor --geo-semi-major
--geosemiminor --geo-semi-minor
--geow1 --geo-w1
--geow2 --geo-w2
--geow3 --geo-w3
--geox --geo-x
--geoy --geo-y
--geoz --geo-z
--halfmaxarea --half-max-area
--halfmaxradius --half-max-radius
--halfmaxsb --half-max-sb
--halfmaxsum --half-max-sum
--halfsumarea --half-sum-area
--halfsumradius --half-sum-radius
--halfsumsb --half-sum-sb
--hostobjid --host-obj-id
--idinhostobj --id-in-host-obj
--magnitudeerr --magnitude-error
--maxvx --max-val-x
--maxvy --max-val-y
--maxvz --max-val-z
--maxx --max-x
--maxy --max-y
--maxz --max-z
--minvx --min-val-x
--minvy --min-val-y
--minvz --min-val-z
--minx --min-x
--miny --min-y
--minz --min-z
--numclumps --num-clumps
--objid --obj-id
--positionangle --position-angle
--riverave --river-mean
--rivernum --river-number
--sberror --sb-error
--semimajor --semi-major
--semiminor --semi-minor
--sigclip-mean-sb-err --sigclip-mean-sb-delta
--skystd --sky-std
--surfacebrightness --sb
--upperlimitmag --upperlimit-mag
--upperlimitonesigma --upperlimit-onesigma
--upperlimitquantile --upperlimit-quantile
--upperlimitsb --upperlimit-sb
--upperlimitsigma --upperlimit-sigma
--upperlimitskew --upperlimit-skew
--weightarea --weight-area
*** MakeNoise
--bgnotmag: new name for the old '--bgisbrightness' option. See the
description of changed '--sum' in MakeCatalog (above) for more.
*** MakeProfiles
--mcolissum: new name for the old '--mcolisbrightness' option. See the
description of changed '--sum' in MakeCatalog (above) for more.
- Units of the output FITS image (value to 'BUNIT' keyword) is now
"counts", until now, it was "brightness". See the description of
changed '--sum' in MakeCatalog (above) for more.
*** Table
- Given the newly added vector columns, the precedence on operations has
been updated. Please read the "Operation precedence in Table" section
of the book.
- To avoid potential loss of information in floating point columns, when
printing the columns to standard output (in the terminal) or saving in
a plain-text file, the default floating point formats are now printed
in exponential (scientific) notation (e.g., 1.234e-5). Until now the
default notation was fixed-point notation (e.g., 0.00001234). A new
sub-section called "Printing floating point numbers" has been added to
the book (under the "Table" section) to clarify this important point.
-A: new short format for --txtf64format. The '-d' short format was
conflicting with the short option name for '--descending'.
*** astscript-psf-select-stars
- Now uses the Gaia DR3 dataset by default (until now it was using eDR3).
*** astscript-psf-stamp
*** astscript-psf-unite
*** astscript-radial-profile
--axis-ratio: new name for old '--axisratio'
--position-angle: new name for old '--positionangle'
*** Library:
- gal_blank_remove_rows: new 'onlydim0' argument to ignore vector columns
when checking for blanks.
- gal_list_str_cat: new 'delimiter' argument to specify the character for
printing between the separate string nodes in the single output string.
- gal_txt_write: new 'tab0_img1' argument. Until now, this function would
distinguish between images and tables using the dimensions of the
input. But with the addition of vector columns in tables (that have 2
dimensions) this argument becomes necessary.
** Bugs fixed
bug #63266: Table ignores a value of 0 given to '--txtf32precision' or
'--txtf32precision=0' (happens when floating point columns
need to be rounded to integers). Reported by Sepideh
Eskandarlou.
bug #63270: Table aborts when called with '--descending' with complaint
on '--txtf64format' (that was added in Gnuastro
0.19). Reported by Giulia Golini.
bug #63285: Table not printing anything in rows of integer columns with
value of zero.
bug #63340: Statistics printing zero for values smaller than 1e-6 and
1e-14 for 32-bit or 64-bit floats respectively. Reported by
Elham Saremi.
bug #63345: WCS coordinate change not accounting for new equinox (only
relevant for equatorial outputs). Reported by Alejandro
Serrano Borlaug.
bug #63399: Table's '--noblank' or '--noblankend' crashing when input
has no rows. Reported by Elham Saremi.
bug #63485: Oversampling in radial profile has incorrect center. Reported
by Nafiseh Sedighi.
bug #63543: Programs delete input when it has same name as output.
Reported by Faezeh Bidjarchian.
bug #63969: make distcheck error due to temporary files of
astscript-psf-select-stars. Reported by Siyang He.
bug #64076: MakeCatalog prints --upnum warning even with
--quiet. Reported by Teet Kuutma.
* Noteworthy changes in release 0.19 (library 17.0.0) (2022-10-24)
** New features
*** Book
- Two new sections added to the "General program usage tutorial" for
describing how to prepare a FITS image for high quality publication (in
PDF), and drawing vector graphics marks from a catalog over it (for
example to show your selected galaxies in the field of view). The use
the newly added features of ConvertType.
*** Arithmetic
- Added new type operators and physical constants. All are also available
in Table's column arithmetic also).
- e: Base of the natural logarithm (no units).
- pi: Fraction of Circle cirumference to diameter (no units).
- c: speed of light in vaccume (in units of m/s).
- G: Gravitational constant (in units of m^3/kg/s^2).
- h: Plank's constant (in units of J/Hz).
- au: Astronomical Units (in units of meters).
- ly: Light years (in units of meters).
- avogadro: Avogadro's constant (in units of 1/mol).
- fine-structure: Fine structure constant (no units).
- counts-to-sb: convert counts to surface brightness (mag/arcsec^2).
- sb-to-counts: convert surface brightness (mag/arcsec^2) to counts.
- mag-to-sb: convert magnitudes to surface brightness over an area.
- sb-to-mag: convert surface brightness to magnitudes over an area.
- New operators that are specific to Arithmetic:
- collapse-median: collapse input dataset by calculating the median
along the given dimension.
- collapse-sigclip-std: Collapse with sigma-clipped standard deviation.
- collapse-sigclip-mean: Collapse with sigma-clipped mean.
- collapse-sigclip-median: Collapse with sigma-clipped median.
- collapse-sigclip-number: Collapse with number remaining after sigma-clip.
*** ConvertType
- It is now possible to draw vector graphics marks from a catalog over
the output PDF images. The following options have been added to
ConvertType for doing this. See the "General program usage tutorial"
for a fully working example.
--marks: name of table containing mark information.
--markshdu: HDU of table if file given to '--marks' is FITS.
--markcoords: name or number of two columns containing coordinates.
--mode: if the coordinates are in 'img' (image) or 'wcs' (RA/Dec).
--markshape: name or number of column containing the shape of each mark.
--markrotate: name or number of column containing rotation of each mark.
--marksize: name or number of column containing the size of the mark.
--sizeinpix: interpret the values in the size column as pixels.
--sizeinarcsec: interpret the values in the size column as arc-seconds.
--sizeinarcmin: interpret the values in the size column as arc-minutes.
--marklinewidth: name or number of column containing mark's line width.
--markcolor: name or number of column containing mark's color.
--listcolors: List all the 140 available colors, and show the colors on
24-bit (true color) terminal.
--marktext: name or number of column containing text under each mark.
--marktextprecision: number of decimals to print as text when the text
column (given to '--marktext') is floating point.
--markfont: name or number of column containing the font to use for the
the mark text (given to '--marktext').
--markfontsize: name or number of column containing the size of the
font to use for the mark text (given to '--marktext').
--showfonts: build a demo PDF with one page per font to show the
various available fonts on the system.
--listfonts: List the names of the available fonts on the terminal.
- It is now possible to select the color of the border of images produced
in vector graphics outputs (EPS and PDF) with '--bordercolor'.
*** Fits
--pixelareaarcsec2: print the image pixel area in units of arcsec^2 to
standard output. Among other things, this is useful in creating a
surface brightness image using the new 'counts-to-sb' operator of
Arithmetic.
--pixelareaonwcs: Ouput an image with the same number of pixels as the
input. But each pixel's value shows its area on the sky (in
degrees-squared). This area is calculated after accounting for
distortion, projection or rotation. Implemented by Pedram
Ashofteh-Ardakani.
--edgesampling: extra sampling along each pixel's edge used to configure
the output of '--pixelareaonwcs', similar to Warp.
*** Statistics
- Linear and Polynomial least squares fitting are now available and very
easy to call on the command-line. They are wrappers over the respective
least squares fitting functions of the GNU Scientific Library. The
interface is pretty simple, like the example below:
aststatistics table.fits -cX,Y,Yerr --fit=linear-weighted
It is also possible to estimate values and errors of the fitted model
on a new X axis. A complete example has been added to the newly added
"Least squares fitting" section of the book (under the Statistics
program documentation). Please see that tutorial to easily use this
feature. The following new options have been added to the Statistics
program for this purpose:
--fit: the model to use. Currently the following models are supported:
linear
linear-weighted
linear-no-constant
linear-no-constant-weighted
polynomial
polynomial-weighted
polynomial-robust
--fitweight: nature of the "weight" column (default: standard dev).
--fitmaxpower: maximum power of X in polynomial models.
--fitrobust: weight function to use in the "robust" polynomial model.
--fitestimate: File name, or number to estimate the fit on.
--fitestimatehdu: HDU containing table in file given to '--fitestimate'.
--fitestimatecol: Column containing X axis values for '--fitestimate'.
*** Table
- It is now possible to customize the format of floating point numbers in
the plain-text outputs: when output is printed on the standard output
(command-line) or in plain-text files. The following new options have
been added for this new feature:
--txtf32format (or '-f'): Format of 32-bit floating point columns. This
can be either 'fixed' (for fixed-point notation) or 'exp' (for
exponential/scientific notation).
--txtf32precision (or '-p'): number of digits following the
decimal-point of 32-bit floating point columns.
--txtf64format (or '-d'): Format of 64-bit floating point columns. This
can be either 'fixed' (for fixed-point notation) or 'exp' (for
exponential/scientific notation).
--txtf32precision (or '-B'): number of digits following the
decimal-point of 32-bit floating point columns.
*** Warp
- Can correct distortions (with any standard recognized by WCSLIB) and
simultaneously align the image to the coordinate system. When no named
linear operation (like '--rotate', '--scale' or etc) is requested, Warp
will go into this mode. It is highly customizable through the following
options. See the "Invoking Warp" section of the book for more. This
feature has been written by Pedram Ashofteh-Ardakani.
--center: RA, DEC of the center of the central pixel of output.
--width: Width of output in degrees or pixels (see '--widthinpix').
--widthinpix: interpret values of '--width' as pixels.
--cdelt: Pixel scale of output ('CDELTi' keywords in FITS).
--ctype: Coordinates and projection algorithm. Default: RA/Dec and
Gnomonic or 'TAN').
--edgesampling: extra sampling of pixel polygon to account for strong
non-linear projection or distortion effects, when necessary.
--gridfile: warp the input to the exact WCS and pixel grid of the file
given to this option. This is very useful when matching images from
differetn surveys. Using this option, you can also insert distortions
in an image (for example on a mock image, to make it match an
observed exposure with dithering+distortion).
--gridhdu: HDU containing image to be matched in '--gridfile'.
--checkmaxfrac: visualize the Moiré pattern of the warp in the second
extension of the output. This is the maximum fraction of a single
input pixel's area in the output pixel. When the output pixel scale
is similar to the input, the Moiré pattern can cause varying
artificial smoothing of the noise level. See the newly added "Moiré
pattern and its correction" section of the book for more on its
basics and how to reduce it in your outputs.
*** astscript-fits-view
--ds9colorbarmulti: show a separate color-bar for each image in DS9. By
default this script will show a single color-bar for all the images to
help save space on the monitor when there are many images.
*** astscript-psf-stamp
- sub-pixel warping is applied to ensure that your coordinate is at the
center of the central pixel of the output image. This results in a
_major_ improvement when estimating the center of the PSF.
--nocentering: disable sub-pixel warping when creating the PSF stamp. As
described above, the sub-pixel warping is critical for the central part
of the PSF, but for the outer parts it is statistically negligible. So
to avoid slowing down you pipeline, you can disable sub-pixel warping
with this option.
--snthresh: if given, the value to this option is assumed to be a
signal-to-noise ratio (S/N) threshold and all pixels below that S/N
will be masked. This is useful because we are often forced to stack
stars of differing magnitudes. While the fainter ones are good for the
inner parts of the star, they degrade the stack's outer parts. With
this option, the fainter stars won't harm the outer parts.
*** GNU Make extensions (in a Makefile)
It is now possible to use custom Gnuastro functions in GNU Make, using
its extension facilities with Dynamic libraries. GNU Make is a very
powerful workflow manager that is also used for data analysis (not just
for compilation). With the Gnaustro Make functions, (astronomical) data
analysis becomes even more easier and faster. In the following, you can
see the first set of such functions (they all begin with 'ast-'). For
more, see the newly added chapter in the Gnuastro manual.
- ast-version-is: will return '1' if the running Gnuastro has the given
version (argument of this function). This can be used to ensure
reproducibility in combination with Make's conditional features, see
the minimal working example in the manual.
- ast-text-contains: will return space-separated words within a larger
list that contain a certain string. The to-contain string can be
anywhere within the words of the larger list.
- ast-text-not-contains: will return space-separated words within a
larger list that DO NOT contain a certain string. The to-not-contain
string can be anywhere within the words of the larger list.
- ast-fits-with-keyvalue: takes a keyword name, a list of keyword
values, a HDU and a list of FITS files. It will return only those
FITS files that have the requested value(s) in the requested keyword
of the requested HDU.
- ast-fits-unique-keyvalues: takes a keyword name, a HDU and a list of
FITS files. It will return all the unique values given to that
keyword within the FITS files.
*** Library:
- GAL_CONFIG_HAVE_PYTHON: non-zero if Python3+Numpy features included.
- GAL_CONFIG_HAVE_GNUMAKE_H: non-zero if GNU Make extensions can be made.
- gal_box_border_rotate_around_center: width of box after rotation.
- gal_color_id_to_name: return the name of a color from its ID.
- gal_color_in_rgb: return the fraction of red-green-blue in a color.
- gal_color_name_to_id: return the ID of a color from its name.
- gal_dimension_collapse_median: collapse input along dim. using median.
- gal_dimension_collapse_sigclip_mean: collapse with sig-clipped mean.
- gal_dimension_collapse_sigclip_std: collapse with sig-clipped STD.
- gal_dimension_collapse_sigclip_median: collapse with sig-clipped median.
- gal_dimension_collapse_sigclip_number: collapse with num. after sig-clip.
- gal_eps_shape_id_to_name: return the name of a shape from its ID.
- gal_eps_shape_name_to_id: return the ID of a shape from its name.
- gal_fit_name_to_id: Convert string name to ID of fitting model.
- gal_fit_name_from_id: Convert ID of fitting model to string name.
- gal_fit_name_robust_to_id: Convert name of robust weights to ID.
- gal_fit_name_robust_from_id: Convert ID of robust weights to name.
- gal_fit_1d_linear: linear fit of input columns.
- gal_fit_1d_linear_no_constant: linear fit with no constant.
- gal_fit_1d_linear_estimate: estimate a linear fit on a new X column.
- gal_fit_1d_polynomial: polynomial fit of input columns.
- gal_fit_1d_polynomial_robust: robust polynomial fit of input columns.
- gal_fit_1d_polynomial_estimate: estimate a polynomial fit on new X column.
- gal_fits_unique_keyvalues: extract all unique values to a certain
keyword in many files.
- gal_fits_with_keyvalue: select FITS image with a certain key value.
- gal_list_data_select_by_name: select a dataset from a list by its name.
- gal_list_str_cat: Concatenate (append) list to a space-separated string.
- gal_list_str_extract: Extract space-separated tokens to a list.
- gal_python_type_from_numpy: Convert Numpy's type id. to Gnuastro's.
- gal_python_type_to_numpy: Convert Gnuastro's type id. to Numpy's.
- gal_txt_contains_string: Check a certain string within in a larger one.
- gal_units_counts_to_sb: SB from counts, zeropoint and area.
- gal_units_mag_to_sb: surface brightness (SB) from magnitude and area.
- gal_units_sb_to_counts: counts from SB, zeropoint and area.
- gal_units_sb_to_mag: magnitude from SB and area.
- gal_warp_pixelarea: return image of same size, but with area on sky.
- gal_warp_wcsalign_init: initialize the WCS aligning structure.
- gal_warp_wcsalign_onpix: Per-pixel filling of output.
- gal_warp_wcsalign_onthread: function to give to pthreads.
- gal_warp_wcsalign: high-level function to align input by its WCS.
- gal_warp_wcsalign_free: free the contents of the WCS aligning structure.
- gal_wcs_free: free a WCS structure that is created or read by Gnuastro.
** Removed features
*** Statistics
--refcol has been removed because it breaks the modularity principle
(given that it is the job of Gnuastro's Table program to limit rows
from a larger table based on many different criteria). The output of
Table can be directly piped to Statistics to achieve the same (and much
more feature-rich effect).
*** Warp
--align: has been removed. This is because aligning an image (while
correcting for any possible distortion) is now the default behavior of
Warp (when no linear operations have been requested).
** Changed features
*** Book
- The "General program usage tutorial" section is now the first section
of the Tutorial chapter, since it introduces the tools at a more basic
level. The "Sufi simulates a detection" (which was previously first)
has been moved to the fourth section.
*** Warp
- The short format of the '--centeroncorner' option has been removed. The
'-c' is now the short format for the new '--center' option to Warp.
*** astscript-psf-scale-factor
--widthinpix: new name for the old '--stampwidth' option. This was done
to have the same name to a similar option in Crop and help in
remembering.
*** astscript-psf-stamp
--widthinpix: new name for the old '--stampwidth' option. This was done
to have the same name to a similar option in Crop and help in
remembering.
*** Library
- gal_eps_write: two new arguments have been added to draw marks, and to
set the border color.
- gal_pdf_write: similar to 'gal_eps_write'.
- gal_fits_hdu_open: new argument to optionally exit program if HDU
couldn't be opened.
- GAL_TABLE_DISPLAY_FMT_FIXED: new name for GAL_TABLE_DISPLAY_FMT_FLOAT
since it corresponds to the fixed-point notation of printing floating
points in plain-text (the '_FLOAT' suffix was too generic and unclear).
** Bugs fixed
bug #62861: '--printkeynames' of Fits program gets caught in an infinite
loop on FITS files that have empty keywords before
'END'. Found by Pedram Ashofteh-Ardakani.
bug #62892: Installed scripts don't account for differing LANG and
LC_NUMERIC. Found by Teet Kuutma and fixed by Raul
Infante-Sainz.
bug #62937: psf-scale-factor not being quiet, when requested. Found and
fixed by Sepideh Eskandarlou.
bug #62943: Couldn't read the value of width with '--snthresh' is called.
Found and fixed by Sepideh Eskandarlou.
bug #62944: No warning when the option value isn't immediately after the
equal sign in long format. Found by Faezeh Bijarchian.
bug #63013: Sigma clip segmentation fault when input has an integer type
with values close to saturation-level.
bug #63022: psf-scale-factor not saving the result in the output file,
found and fixed by Raul Infante-Sainz.
bug #63189: MakeProfiles custom profiles become NaN with a single row
being NaN, reported by Nafise Sedighi.
bug #63207: Match crashes when one input has no rows. Found by Sepideh
Eskandarlou.
bug #63257: Fits program's '--skycoverage' gives unreasonable outputs
when image crosses the RA=0 hour circle. Found by Irene
Pintos Castro.
bug #63261: Radial profile script ignores central pixel in azimuthal
profiles. Found and fixed by Sepideh Eskandarlou.
* Noteworthy changes in release 0.18 (library 16.0.0) (2022-07-21)
** New features
*** Book
- New "Measuring the dataset limits" section has been added in second
tutorial. Using the catalogs produced during the analysis, it shows
how to derive a dataset's magnitude limit, completeness limit
(crudely: without simulations), and surface brightness limit. All are
important measures in any scientific analysis! This section was
written with the help of S. Zahra Hosseini Shahisavandi and Sepideh
Eskandarlou.
*** Arithmetic
- New operators (also available in Table's column arithmetic):
- stitch: connect any number of input images along the given dimension
(for example to stitch the images of separate amplifiers of a CCD
into one image).
- noblank: Remove blank elements from the input and return a 1D output
(irrespective of how many dimensions the input had).
- jy-to-counts: convert Janskys to counts using an AB-magnitude-based
Zeropoint (the inverse operator 'counts-to-jy' already existed).
- mag-to-jy: convert AB magnitudes to Janskys.
- jy-to-mag: convert Janskys to AB magnitudes.
- load-col-%-from-%-hdu-%: Load a certain column into the operands
stack. The name or number of the column is the first '%'. The file
name hosting that column is the second '%', and HDU (if the file is
a FITS file), is identified by third '%'. This can be used to load
columns outside the main table in Table's column arithmetic or by
operators that need columns as input parameters (like the new
'random-from-hist' operator).
- random-from-hist-raw: return a set of random points based on a
custom distribution defined by its histogram. See the description
and complete examples in the book for more.
- random-from-hist: Similar to 'random-from-hist-raw', but within each
bin, the final returned value is itself the result of a uniform
sampling (to make the final distribution less descrete). See the
example in the book for selecting random star magnitudes based on
Gaia's observed magnitude distribution.
- New options to add metadata (standard FITS keywords) to your output
for easy understanding of the data in the future, or by your
colleagues if you share the output). Until now, to have this feature,
you needed to call Fits on the output of Arithmetic (for setting the
HDU name to "my-sum" for example):
astarithmetic img-a.fits img-b.fits + --output=sum.fits
astfits sum.fits --write=EXTNAME,"my-sum"
But with the following commands, you can add important metadata to the
output of Arithmetic in the same command and simplify your scripts:
--metaname: Metadata name of the output (FITS keyword: 'EXTNAME').
--metaunit: Unit of the output's pixels (FITS keyword: 'BUNIT').
--metacomment: Description of the data (FITS keyword: 'COMMENT').
Like the following for the example above:
astarithmetic img-a.fits img-b.fits + --metaname="my-sum" \
--output=sum.fits
*** Crop
--oneelemstdout: when a crop has a single pixel and this option is
called, the single pixel's value will be printed on the standard
output instead of creating a FITS file. This option can be useful in
labeled images like detection maps (when you want to see what label a
certain coordiante/pixel corresponds to). This option was suggested by
Raul Infante-Sainz.
*** Fits
- When no option is specified (thus printing the HDU metadata in the
given file), the units of the HDUs are also printed as a new
column. For tables, it will be blank ('n/a'); because tables have
separate units on each column (use 'asttable table.fits -i').
--copykeys: can now take any number of keyword names also. For example
'--copykeys=KEYNAME1,KEYNAME2,KEYNAME3'. Until now, it was only
possible to give this option a range of keyword positions (for example
'--copykeys=10:13' to copy the 10th to 13th keywords (inclusive). But
now, you can also select non-contiguous set of keywords to copy from
one HDU into another HDU (possibly in another file). This was
suggested by Juan Antonio Fernández Ontiveros and implemented with the
help of Jash Shah.
*** MakeProfiles
- It is now possible to insert a custom image as a new (10th)
'custom-img' profile. This is useful when you have no-noise FITS
images from numerical simulations for example, and you want to insert
those at a certain location within a larger image (while optionally
adding existing Sersic, point or other profiles in the same
command). The magnitude of the inserted image can be set in the
magnitude column of the input catalog. This can be disabled (to use
the original image pixel values) using the '--mcolnocustimg' option.
--mcolnocustprof: new option to disable scaling of custom radial profiles
to required magnitude when building a 2D image from them.
*** Query
- Short name for Gaia's DR3 dataset has been added for both ESA's Gaia
database and VizieR, for example with the following commands:
astquery gaia --dataset=dr3 --center=1.234,5.678 --radius=0.01
astquery vizier --dataset=gaiadr3 --center=1.234,5.678 --radius=0.01
*** astscript-radial-profile
--zeroisnotblank: new option to account for zero-valued pixels in the
profile measurement. By default, during the internal cropping phase,
zero-valued pixels are set to blank (and thus not considered during
the profile measurement). When this option is called, it is directly
passed to the internal cropping command ('astcrop' has an option with
the same name). This was suggested by Ignacio Ruiz Cejudo and
implemented by Sepideh Eskandarlou.
*** Library
- GAL_ARITHMETIC_OP_UNIQUE: to find unique elements in dataset.
- GAL_ARITHMETIC_OP_NOBLANK: to remove blank elements from dataset.
- GAL_ARITHMETIC_OP_STITCH: to stitch multiple datasets.
- GAL_ARITHMETIC_OP_RANDOM_FROM_HIST: the new 'random-from-hist' operator.
- GAL_ARITHMETIC_OP_RANDOM_FROM_HIST_RAW: the new 'random-from-hist-raw' op.
- gal_arithmetic_load_col: Low-level function for parsing the string of
the newly added 'load-col' operator, mentioned in the Arithmetic
section above.
- gal_statistics_std_from_sums: return the standard deviation using the
sum, sum of squares and number of a distribution.
** Removed features
** Changed features
*** MakeCatalog
--std: measures the standard deviation of the object/clump pixels of the
values image. Until now, there was no column to calculate this,
and the '--std' column returned the _sky_ standard deviation over
the object/clump labels (from the standard deviation image,
ignoring the values image). But that was confusing, especially
since we have columns like '--mean' or '--median' which only use
the values image (see bug #62685).
--skystd: new name for old '--std', see description of '--std' above.
--sfmagnsigma: default value changed to 3 (more commonly used).
--sfmagarea: default value set to 100 arcsec^2 (more commonly used).
--upnum: MakeCatalog won't abort when '--upnum' is too small (less than
20). '--upnum' defines the number of random positions for upper-limit
measurements, so when its too small, the statistical precision of the
result will badly affected and MakeCatalog would abort with an error
message. However, in some scenarios (for example when the positions of
the random apertures from '--checkuplim' are needed) the actual
upper-limit measurements aren't relevant! So from this version,
instead of aborting MakeCatalog, it prints a warning and
continues. This was suggested by S. Zahra Hosseini Shahisavandi and
Ignacio Trujillo.
*** MakeProfiles
- The string identifier for custom radial profiles is now called
'custom-prof' (until now, it was called 'custom'). This was necessary
because of the new custom image feature that has been added. The code
for this custom radial profile function (8) hasn't changed.
- After constructing a custom radial profile (the 'custom-prof'
profile), MakeProfiles will scale it to have the desired total
magnitude in the input catalog of all profiles. Until now, the 2D
images created from a custom radial profile wouldn't be touched and
the pixel values would exactly come from the custom radial profile
(ignoring the magnitude column of the input catalog). To have the old
behavior, use the new '--mcolnocustprof' option. This new
functionality was proposed by Elham Saremi.
*** Table
--catcolumnfile: the '-N' suffix (where 'N' is a counter for the files
with concatenated, or appended, column) is only added if a column with
a identical column name (not case sensitive) existed in the
table. Until now, this suffix would be added to all the newly
concatenated columns (recall that the suffix addition can be disabled
in general with the '--catcolumnrawname' option).
--rowrange: new name for the old '--rowlimit'. This option takes the
range of desired rows based on their position (for example
'--rowrange=2,5' will only print rows 2, 3, 4 and 5 of the table). But
the old name was very confusing and would not directly convey this
behavior.
*** astscript-radial-profile
--azimuth: if the first azimuthal angle is larger than the second, the
outer region between the two angles will be used. For example
'--azimuth=80,40' will use the full azimuthal range except for the
region with azimuthal angles between 40 and 80 degrees. This can be
useful to ignore parts of the profile. See the example in the
description of this option on how to visually check the pixels used
for the profile.
** Bugs fixed
bug #62216: MakeProfiles crash when a 3D cube is requested and input
catalog is from a pipe. Found by Irene Pintos Castro.
bug #62220: astscript-psf-stamp crashes when coordinate not in image.
Found and fixed by Sepideh Eskandarlou.
bug #62250: Convolve crashes when doing PSF matching. Found with help of
Raul Infante-Sainz.
bug #62252: Radial profile giving one larger radius in high axis-ratio
profile. Found and fixed by Sepideh Eskandarlou.
bug #62253: Arithmetic: segmentation fault when invalid input given for
'ra-to-degree' or 'deg-to-degree' operators. Found by Pedram
Ashofteh-Ardakani.
bug #62305: Not reading the 'MemFree' keyword in '/proc/meminfo' to find
the available RAM (and thus printing an annoying
warning). Reported by Juan Miro.
bug #62309: MakeCatalog segmentation fault when clumps image given as
main input (objects should be given). Reported by Sepideh
Eskandarlou.
bug #62429: Segment prints one line status-progress line when called with
'--quiet'. Reported by Teet Kuutma.
bug #62548: Table doesn't recognize plain-text string column when defined
width is larger and it is the last column. Found with the
help of Sepideh Eskandarlou.
bug #62563: Some checks of installed scripts get skipped on macOS. Found
and fixed by Raul Infante-Sainz.
bug #62564: Arithmetic not parsing numbers in scientific notation (for
example '1e5'). Found by Elham Saremi.
bug #62590: Failure to build on macOS when building in debugging mode
('./configure --enable-debug'). Found by Raul Infante-Sainz.
bug #62597: Arithmetic not writing single-valued output into a file when
called with '--output'. Reported by Raul Infante-Sainz.
bug #62636: Table ignoring '--range' when '--head' is also
called. Reported by Sepideh Eskandarlou.
bug #62662: Table crashes when column arithmetic is applied after adding
rows from another file (with '--catrowfile').
bug #62674: Column arithmetic can't use columns from --catcolumnfile.
bug #62678: Radial profile script using values image instead of STD image
when '--instd' not used. Found and fixed by Raul Infante-Sainz.
bug #62679: MakeProfiles kernel only normalized when zeropoint is
zero. Found by Raul Infante-Sainz.
bug #62680: MakeProfiles using WCS-options when --background is given and
the background doesn't have WCS. Found with the help of Raul
Infante-Sainz.
bug #62683: Arithmetic's 'set-' operator not working after the 'makenew'
operator. Found by Raul Infante-Sainz.
bug #62685: MakeCatalog's '--std' option is wrongly interpreted as the
standard deviation of the values image, while it is actually
the root mean square of the Sky standard deviation
pixels. Found by Raul Infante-Sainz.
bug #62694: Radial profile script overestimates the surface brightness
error. Found and fixed with the help of Raul Infante-Sainz.
bug #62710: Plain-text integers starting with 0 are read in the octal
base (which is common in software engineering, but not in
data analysis). Found by Manuel Sánchez-Benavente.
bug #62718: Table goes into an infinite loop on some old ASCII FITS
tables. Found by Hilderic Browne.
bug #62720: Table not reading string columns that are shorter than the
defined width in the metadata.
bug #62721: Radial profile script can't deal with angles around azimuth
zero (for example '--azimuth=355,5' to get only the azimuthal
range of 10 degrees around the major axis). Found and fixed
by Samane Raji.
bug #62784: Conversion of sexagesimal RA or Dec ignores sign when first
digit is zero (for example '-00d12m34'). Found by Manuel
Sánchez-Benavente.
bug #62794: psf-stamp scripts produces inconsistant outputs on
failure. Found by Sepideh Eskandarlou and Nafise Sedighi.
* Noteworthy changes in release 0.17 (library 15.0.0) (2022-03-20)
** New features
New programs:
- astscript-fits-view: Given any number of FITS files, this script will
either open SAO DS9 (for 2D images or 3D cubes) or TOPCAT (for tables)
to visualize their contents in a graphic user interface (GUI). Both
SAO DS9 and TOPCAT are free software packages that are heavily used in
astrophysics. When opening in DS9, files will be opened as "match"ed
and "lock"ed frames, enabling easy visual comparison. For GUIs that
conform to the freedesktop.org standards, a '.desktop' file is also
provided that you can easily activate (as described in the
manual). Once activated, simply clicking on FITS file(s) in your
graphic environment will open DS9 or TOPCAT depending on their
contents. This script was suggested and partly written by Sepideh
Eskandarlou.
- A set of installed scripts to enable easy estimation and subtraction
of the extended PSF (in a highly modular and easily scalable
manner). They are primarily based on the method described in
Infante-Sainz et al. 2020
(https://ui.adsabs.harvard.edu/abs/2020MNRAS.491.5317I) and primarily
maintained by Raul Infante-Sainz himself and the help of Sepideh
Eskandarlou. A full tutorial on real data from the J-PLUS survey
(http://j-plus.es) has also been added to the tutorials chapter to
describe in full detail how to run these scripts in a generic way (on
any survey). These installed scripts all have a 'astscript-psf-*'
prefix in their names:
- astscript-psf-select-stars: Find all the stars within an image
that are suitable for constructing an extended PSF. If the image
has WCS, this script can automatically query Gaia to find the
good stars.
- astscript-psf-stamp: build a crop (stamp) of a certain width
around a star at a certain coordinate in a larger image. This
script will do sub-pixel re-positioning to make sure the star is
centered and can optionally mask all other background sources).
- astscript-psf-scale-factor: Given a PSF model, and the central
coordinates of a star in an image, find the scale factor that has
to be multiplied by the PSF to scale it to that star.
- astscript-psf-unite: Unite the various components of a PSF into
one. Because of saturation and non-linearity, to get a good
estimate of the extended PSF, its necessary to construct various
parts from different magnitude ranges.
- astscript-psf-subtract: Given the model of a PSF and the central
coordinates of a star in the image, do sub-pixel re-positioning
of the PSF, scale it to the star and subtract it from the image.
Book:
- New tutorial on how to extract the extended Point Spread Function
(PSF) from the actual science data and how to subtract
it. Written with the help of Raul Infante-Sainz and Sepideh
Eskandarlou.
- New section called "Skewness cased by signal and its measurement"
added to the third tutorial (Detecting large extended targets). It
describes the significant advantages of using the quantile of the mean
as a measure of skewness and why its a very important factor when
understanding how NoiseChisel works. It was originally written by
Sepideh Eskandarlou, with edits by Elham Saremi and Pedram
Ashofteh-Ardakani.
- New section called "Integer benefits and pitfalls" added under the
Arithmetic program's documentation. It describes the running time,
storage and RAM consumption benefits if you use integers (where
possible), and the issues/solutions you may confront when doing so.
All programs:
- Coordinate-related columns in all programs now also accept sexagesimal
values, not just degrees. Therefore, '--center=113.8729761,31.9027152'
and '--center=07h35m29.51,31d54m9.77' are now equivalent in the Crop
or Query programs that have this option! Other improved options
include the '--polygon' options of Table and Crop (that also take
region files produced by DS9), the '--coord' option of Match, the
'--crval' option of MakeProfiles.
- Columns of FITS tables are now read in parallel (when '--numthreads'
isn't set to '1', and if the dependency CFITSIO library was built with
'--enable-reentrant'). In scenarios where many columns may be
necessary (for example the Table or Match programs), this will greatly
improve the running time of the programs. This task was implemented
based on feedback from Andrés Del Pino Molina.
Arithmetic:
- New operators:
- add-dimension-fast: Add a new dataset along the "fastest" dimension
of the first dataset (in a FITS image, the "fast" dimension is the
horizontal one). For example if you have N one-dimensional datasets
with M elements, you can use this operator to construct a
2-dimensional FITS image that has N pixels along the horizontal and
M pixels along the vertical.
- repeat: Put a copy of the given dataset onto the top of the stack of
operators for the requested number of times.
Crop:
--widthinpix: when in WCS-mode, the value given to '--width' will be
interpreted in units of pixels. This will allow having crops of a
fixed pixel size, even when the mode of the central coordinate is in
WCS. This was suggested by Jesús Varela.
Fits:
- If a 'CHECKSUM' keyword exists in a HDU and any of the keyword
modification keywords are called, Fits will automatically update the
checksum after all the changes are done. This is important to keep the
checksum valid after the change (recall that you can verify the FITS
checksums with the '--verify' option to Fits). This task was suggested
by Tamara Civera Lorenzo.
Table:
- Options that accept strings ('--colmetadata', '--equal' and
'--notequal') can now accept a comma within the string: to avoid
confusing the comma with a separator of values, you should put a '\'
before it. For example '--equal=AB,cd\,ef' will select all rows where
the 'AB' column has a value of 'cd,ef'. This task was suggested by
Zohreh Ghaffari.
--catrowfile: File to concatenate (i.e., add or append) rows into the
main input table. With this option, you can add the rows of another
table into the final output. This option can be called multiple times,
allowing you to merge rows of any number of tables. This job is done
after concatenating columns, but before any of the row-selection
operations. This was suggested by Raúl Infante-Sainz and Manuel
Sánchez Benavente.
--catrowhdu: The HDU(s) of the FITS file(s) given to '--catrowfile'.
--noblankend: remove blank rows from selected columns of the _output_
table, for example after after adding columns by column arithmetic or
from other tables (this is the new name for the old '--noblank'
option, see the changed features below). Like the new '--noblank',
this option can also be called multiple times, so '--noblankend=1
--noblankend=2' is equivalent to '--noblankend=1,2'.
- New operator for column arithmetic:
- date-to-millisec: convert the formatted date string into the number
of milli-seconds (as a 64-bit integer) since Unix-time (00:00:00 of
1970-01-01).
Match:
- k-d tree based matching has been added for finding the matching rows,
and is set to the default mode. It will do the matching in parallel,
which will improve the speed when many rows are present. The k-d tree
can also optionally be saved in a file for matching new tables with
the same one (further improving running speed). For a full description
of Match's new behavior, please see the new "Matching algorithms"
section of the book. This feature was implemented with the help of
Sachin Kumar Singh.
--kdtree: new option to specify the algorithm used by Match. It can take
four values: 'internal' (which is the default), 'build' (to build a
k-d tree), 'FILE.fits' (name of k-d tree to import), or 'disable' (to
do the matching with the old sort-based method). For more, please see
the new "Matching algorithms" section of the book.
--kdtreehdu: the HDU of the external k-d tree (if a FITS file was given
to '--kdtree').
Library:
- gal_fits_key_exists_fptr: Check if a certain keyword exists in the
already-open FITS file pointer.
** Removed features
Nothing was removed in this release.
** Changed features
Arithmetic:
- 'add-dimension-slow' is the new name for the old 'add-dimension'.
Segment:
- '--noobjects' is the new name for the old '--onlyclumps' option. This
is because the name Only-Clumps can lead to a wrong expectation that
the output will _only_ contain a 'CLUMPS' HDU. However,
with'--onlyclumps', the input image (without a Sky level) and the Sky
standard deviation image are also present as HDUs in the output. To
only get the 'CLUMPS' HDU, it is necessary to also use the
'--rawoutput' option. This renaming was done to avoid such confusions
and was raised by Sepideh Eskandarlou.
Match:
- By default (when '--quiet' isn't called), Match will print the names
of the inputs and timings of important steps to the standard output
(on the command-line).
Table:
- The '--noblank' option's logic is very similar to other row-selection
(by value) options (like '--range', '--polygon' or
'--equal'). However, until now, it was applied at the end of Table,
just before writing the output (so it would apply to the output table,
that may have new/different column names/orders, not the input table
like the other similar options). This was very inconvenient and
non-intuitive. From this version, '--noblank' option now behaves
similarly to the other input row-selection (by value) options. So it
can be called multiple times (to give many columns in multiple calls)
and it can be called for a column that is not necessarily one of the
output columns (for example you want the RA and Dec of all rows
without a blank magnitude, so you don't need to have the magnitude
column any more!). The old behavior of this option is now available in
the new '--noblankend' option. This point was raised by Samane Raji.
- 'date-to-sec' operator now only returns the number of seconds and is
always in a 64-bit signed integer format. Until now, if the input had
sub-second precision it would return a 64-bit floating point type;
however, that resulted in bug #61976 (where the loss of precision due
to the floating-point's logarithmic scale for too many significant
digits caused incorrect conversions).
Library:
- gal_fits_tab_read: now takes the number of threads to read the desired
columns in parallel (this will greatly improve reading speed when
there are many columns and many rows).
- gal_table_read: similar to 'gal_fits_tab_read' (only relevant for
reading FITS tables).
- gal_blank_initialize: also works on string data, but initializing only
a tile over a larger block of elements is only supported for numeric
data types.
** Bugs fixed
bug #61329: make check crash in macOS in convolve/spectrum-1d.sh, found
and fixed with the help of Sebastian Luna-Valero.
bug #61378: Table crash when input is sexagesimal RA/DEC and no metadata.
bug #61462: Crash on FITS binary table columns with repeat of 0.
bug #61468: DS9 region file can't be read in sexagesimal coordinates,
reported by Juan Antonio Fernández Ontiveros and Alejandro
Lumbreras Calle.
bug #61490: Extra columns in the metadata causes crash in plain-text
tables, found by Zohreh Ghaffari.
bug #61493: NoiseChisel's --checksn not showing pseudos equal to
--snminarea (only larger ones), found by Raúl Infante-Sainz.
bug #61584: When FITS file doesn't end with '.fits' and the output file
already existed, Arithmetic would write the output in new
extension while it should have been deleted.
bug #61598: ds9-region script not using precise WCS values, found and
fixed by Raúl Infante-Sainz.
bug #61698: Mean operator of Arithmetic (for stacking many images) not
returning NaN for blank regions when input is integer. This
fix also resolves the same problem in the 'sum', 'std',
'median' and all 'sigclip-*' operators); reported by Zohreh
Ghaffari.
bug #61740: WCSLIB conflicting with system's wide-character string (WCS)
library in Gnulib checks for some systems; reported by Martin
Guerrero Roncel.
bug #61940: Numbers with many decimals (like '330624.3918430004', common
in Julian dates for example) incorrectly read as float32
(thus loosing precision); reported by Zohreh Ghaffari, fixed
with help of Pedram Ashofteh-Ardakani.
bug #61967: DS9 polygon region files not read when they have width and
color; reported by Zohreh Ghaffari, fixed by Pedram
Ashofteh-Ardakani.
bug #61976: Table arithmetic date-to-sec produces same result for
different times (separated by about 1 second). This bug was
reported by Zohreh Ghaffari and Pedram Ashofteh-Ardakani.
bug #62008: Arithmetic not deleting existing output when the 'makenew' is
used (no FITS file exists in the reverse polish notation).
bug #62052: WCS decomposition of CD into PC+CDELT not setting internal
values; reported by Ignacio Ruiz Cejudo and Raul
Infante-Sainz.
bug #62054: Crash in Table's '--catrowfile' when string column is present;
reported by Manuel Sánchez-Benavente.
bug #62069: Wrong Arithmetic result in binary operators when both input
operands are integers of same width, but different sign (for
example 'int32' and 'uint32'); reported by Zohreh Ghaffari
and Pedram Ashofteh-Ardakani.
bug #62070: Segmentation fault in Table's '--tail' option, when string
columns are requested and the requested number of rows given
to '--tail' is more than half of the number of rows; reported
by Manuel Sánchez-Benavente.
bug #62096: 'astarithmetic 250 1 +' not producing correct result;
reported by Raul Infante-Sainz.
bug #62112: NoiseChisel crash when '--checktiles' and
'--continueaftercheck' called together; reported by Giulia
Golini.
bug #62127: GNU Scientific Library's 1D Steffen interpolation method
(that guarantees monotonicity) not used in Gnuastro's
library, even if it is present and found at configure time;
found and fixed by Pedram Ashofteh-Ardakani.
bug #62204: Column arithmetic crashes when there are no rows; reported by
Zohreh Ghaffari.
* Noteworthy changes in release 0.16 (library 14.0.0) (2021-10-10) [stable]
** New features
Book:
- New section added to ConvertType's documentation for annotating the
PDF output images with coordinates on the edges of the images, or
tangential size at certain redshift. This is done using the PGFPlots
package of LaTeX, directly with the same fonts and graphical settings
as your paper/slide, while being in vector graphics quality, and not
requiring hundreds of dependencies (only LaTeX is necessary beyond
Gnuastro).
- New section to describe the precedence of Table operators in one call
to Table. Since the number of operations on table columns/rows has
greatly increased in the last few versions, knowing this precedence
can minimize the number of calls to Table.
- The section on reverse polish notation (used in Arithmetic, for
images, and column arithmetic in Table) has been edited and extended
to be more useful for a new user.
- Due to the great expansion of the number of operators in the
Arithmetic program and Column Arithmetic in the table program, the
"Arithmetic operators" section of the book has now grouped similar
operators in sub-sections to greatly help in finding a certain option.
Arithmetic:
- New operands (also available in Table's column arithmetic, the
distance conversion operators were suggested by Markus Schaney):
- box-around-ellipse: width and height of the box covering an ellipse.
- au-to-pc: Convert Astronomical Units (AUs) to Parsecs (PCs).
- pc-to-au: Convert Parsecs (PCs) to Astronomical Units (AUs).
- ly-to-pc: Convert Light-years (LYs) to Parsecs (PCs).
- pc-to-ly: Convert Parsecs (PCs) to Light-years (LYs).
- ly-to-au: Convert Light-years (LYs) to Astronomical Units (AUs).
- au-to-ly: Convert Astronomical Units (AUs) to Light-years (LYs).
CosmicCalculator:
- A warning is printed if the requested redshift is lower than 0.007
(corresponding to ~30Mpc in Plank 2018 cosmology). Because at these
scales the peculiar velocity of galaxies may be significant compared
to the Hubble flow (which is the basis of the calculations here). The
warning can be suppressed with the '--quiet' option. This was
suggested by Ignacio Trujillo.
MakeCatalog:
--areaerror: spatial resolution of image specified by user, used in
estimating the surface brightness error.
--sberror: error in measuring the surface brightness (mag/arcsec^2).
MakeProfiles:
- New type of profile showing the azimuthal angle (in degrees, along the
elliptical circumference of fixed radius) of each pixel. In
combination with the radial distance profile, you can now create
complex features in polar coordinates, such as tidal tails or tidal
shocks (using the Arithmetic program to mix the radius and azimuthal
angle through a function to create your desired features). Implemented
after discussions with Fernando Buitrago and Matthias Kluge.
Match:
- When called with '--notmatched --outcols=AAA,BBB', Match will append
non-matching rows of second table into first table's rows (for columns
'AAA' and 'BBB' in example above, which are assumed to exist in both
inputs, with same numeric datatype). This allows easy/clean merging of
two catalogs that may have matching objects into one catalog without
repetition. See description of '--outcols' in the Match manual for
more. This feature was proposed by Juan Molina Tobar and Leslie Hunt.
Sort-by-night (installed script)
--stdintimeout: new option to set internal delay time for input from
pipes (standard input). This is not for inputs to the script (which
should always be files), but for the script's internal pipes.
Statistics:
--quantofmean: the quantile of the mean of the input dataset. this is a
very good statistic to measure skewness in a distribution, see the
description of this option in the book for more.
Library:
- Arithmetic macros:
- GAL_ARITHMETIC_OP_BOX_AROUND_ELLIPSE
- gal_wcs_write_wcsstr: wrapper of wcshdo of WCSLIB to simplify generic
access to that low-level operation (has some extra sanity checks).
- gal_units_au_to_pc: Convert Astronomical Units to Parsecs.
- gal_units_pc_to_au: Convert Parsecs to Astronomical Units.
- gal_units_ly_to_pc: Convert Light-years to Parsecs.
- gal_units_pc_to_ly: Convert Parsecs to Light-years.
- gal_units_ly_to_au: Convert Light-years to Astronomical Units.
- gal_units_au_to_ly: Convert Astronomical Units to Light-years.
** Removed features
** Changed features
All programs:
--stdintimeout: default value changed from 0.1 seconds to 1.5 seconds to
avoid too many crashes when command before the pipe takes longer (thus
needing a manual change of this value).
MakeProfiles:
- The default output suffix (when no '--output' is given) is now
'_profiles.fits'. Until now, when no output name was given,
MakeProfiles just replaced the suffix with '.fits' (effectively
assuming the input is a plain-text file). As a result, if the input
catalog was a FITS file, it would be automatically written-over
(causing an error: bug #60989).
Match:
- The two '--notmatched' and '--outcols' can be called together (to
create a single catalog that appends the non-matching rows of second
table with the rows of the first. Until this version, this would cause
an error.
Table
- Column concatenation (adding columns from other files) is now done
before all row selection or row re-ording operations (if
called). Until now, adding columns from other files was done after all
row selection or re-ordering, but this was not too practically useful.
- Column arithmetic operators:
- 'wcs-to-img' new name for old 'wcstoimg' operator. The new
unit-conversion operators generally have a '-to-' between the units
to help in readability. So to avoid confusion, this operator has
also been changed.
- 'img-to-wcs' new name for old 'imgtowcs' operator (see description of
'wcs-to-img' above).
Library
- gal_fits_key_write_filename: now has 'quiet' argument to avoid
printing a warning.
** Bugs fixed
bug #60725: MakeCatalog doesn't put comment on --halfsumsb column.
bug #60776: Radial profile script not using standard deviation image,
fixed by Zahra Sharbaf.
bug #60778: Brightness error not NaN when all STD pixels are blank,
reported by Zahra Sharbaf.
bug #60826: Arithmetic won't delete existing file with tofile operators.
bug #60881: Query segmentation fault when NED is called without a dataset.
bug #60901: sort-by-night crash during make check due to stdintimeout,
reported by Zahra Sharbaf.
bug #60903: Increasing value of stdintimeout not effective beyond 1000000,
reported by Zahra Sharbaf.
bug #60909: WCS coordinate conversion not working with TPV distortion,
fixed with the help of Mark Calabretta.
bug #60923: Standard deviation calculation giving NaN values due to
floating point errors, found and fixed by Natáli Anzanello.
bug #60958: Warning on long filename not written in FITS keyword printed
even when '--quiet' is called, found by Raúl Infante-Sainz.
bug #60989: MakeProfiles deletes input fits catalog when no output name
specified.
bug #60999: No content check when plain-text table doesn't have metadata.
bug #61007: Crash due to CFITSIO 4.0.0 version format change, reported by
Vincenzo Testa and Zohreh Ghaffari.
bug #61108: Plain text table metadata ignored when numeric data type was
not understood, reported by Sepideh Eskandarlou.
bug #61128: Radial profile: --instd and --stdhdu error when space used
before value, found and fixed by Sepideh Eskandarlou.
bug #61192: BuildProgram custom program linking problems on Debian-based
OSs, reported by Raúl Infante-Sainz.
bug #61244: Sum operator in arithmetic ignoring floating point images
with blank values, found and reported by Giulia Golini and
Raúl Infante-Sainz.
bug #61287: Two "switch" blocks miss "break" in MakeCatalog when setting
output column information, found and fixed by Vladimir
Markelov.
bug #61293: MakeCatalog not properly initializing the '--minvy' column
when parsing clumps, found and fixed by Vladimir Markelov.
* Noteworthy changes in release 0.15 (library 13.0.0) (2021-05-30) [stable]
** New features
New programs:
- astscript-ds9-region: Given a table (either as a file or from
standard input), create an SAO DS9 region file from the requested
positional columns (WCS or image coordinates). For example with the
command below you can select certain rows of a given table, and show
them over an image:
asttable table.fits --range=MAGNITUDE,18:20 --column=RA,DEC \
| astscript-ds9-region --column=1,2 --radius=0.5 \
--command="ds9 image.fits"
This script was written with the help of Samane Raji.
- astscript-radial-profile: Measure the radial profile of an object on
an image. The profile can be centered anywhere in the image and any
circular, or elliptical distance can be defined. The output is a table
with the profile's value in one column and any requested measure in
the other columns (any MakeCatalog measurement is possible). This
script was written and is maintained by Raúl Infante Sainz. Zahra
Sharbaf and Carlos Morales Socorro also contributed.
All programs:
- FITS files that don't have a recognized FITS suffix (e.g., '.fits',
'.fit', '.fits.gz', etc) will also be recognized. Generally, it is
good practice to keep a suffix to be more human and computer friendly
(checking three or four characters is much faster than opening the
file and checking the contents to conform with the FITS standard), but
when you are given such a file (by someone else), this new feature can
be handy. Suggested by Clotilde Laigle and Leigh Smith.
--wcslinearmatrix: new option in all programs that lets you select the
output WCS linear matrix format. It takes one of two values: 'pc' (for
the 'PCi_j' formalism) and 'cd' (for 'CDi_j'). Until now, the outputs
were always stored in the 'PCi_j' formalism (which is still the
recommended format).
Book:
- New "Image surface brightness limit" section added to the third
tutorial (on "Detecting large extended targets"). It describes the
different ways to measure a dataset's surface brightness limit and
upper-limit surface brightness, while discussing their
differences. This tutorial was written with the help of Ignacio
Trujillo and Raúl Infante Sainz.
Arithmetic:
- New operators (all also available in Table's column
arithmetic).
- sin: Trigonometric sine (input in degrees).
- cos: Trigonometric cosine (input in degrees).
- tan: Trigonometric tangent (input in degrees).
- asin: Inverse of trigonometric sine (output in degrees).
- acos: Inverse of trigonometric cosine (output in degrees).
- atab: Inverse of trigonometric tangent (output in degrees).
- sinh: Hyperbolic sine.
- cosh: Hyperbolic cosine.
- tanh: Hyperbolic tangent.
- asinh: Inverse of hyperbolic sine.
- acosh: Inverse of hyperbolic cosine.
- atanh: Inverse of hyperbolic tangent.
- mknoise-sigma: Add Gaussian noise with the fixed sigma.
- mknoise-poisson: Add Poisson noise with the given background.
- mknoise-uniform: Add uniform noise around existing value.
- counts-to-mag: Convert counts to magnitudes with given zero point.
- mag-to-counts: Convert magnitudes to counts with given zero point.
- counts-to-jy: Convert counts to Janskys through a zero point based
on AB magnitudes.
Trigonometric operators proposed by Zahra Sharbaf and Samane Raji.
--envseed: new option to get random number generator settings for the
new 'mknoise-sigma' and 'mknoise-poisson' operators from the
environment for reproducibility (see "Generating random numbers"
section in manual).
ConvertType:
--globalhdu: Use a single HDU identifier for all the input files
files. Its operation is identical to the similarly named option in
Arithmetic. Until now it was necessary to call '--hdu' three times if
you had three input FITS files with input in the same HDU.
Crop:
--polygon: now also accepts an SAO DS9 region file as input (as well as
coordinates). This enables you to draw your polygon over the image
through the GUI interface of SAO DS9, save your polygon as a "region"
file, and feed the region file into Crop directly. Until now, the only
way to define the polygon was to give all the polygon vertice
coordinates, one-by-one to the '--polygon' option (which was tedious
and buggy), but is still possible. This was implemented by Natáli
Anzanello.
Fits:
--hastablehdu: print 1 if at least one table HDU exists in file.
--hasimagehdu: print 1 if at least one image HDU exists in file.
--listallhdus: print all HDU names (or numbers when no name exists).
--listtablehdus: print table HDU names (or numbers when no name exists).
--listimagehdus: print image HDU names (or numbers when no name exists).
--printkeynames: Print all keyword names in current HDU.
--wcscoordsys: convert the WCS coordinate system of the input into any
recognized coordinate system. It currently supports: equatorial
(J2000, B1950), ecliptic (J2000, B1950), Galactic and
Supergalactic. For example if 'image.fits' is in galactic coordinates,
you can use this command to convert its WCS to equatorial (J2000):
astfits image.fits --wcscoordsys=eq-j2000
This option only works with WCSLIB 7.5 and above (released in March
2021), otherwise it will abort with an informative warning. This was
done with the help of Mark Calabretta.
--keyvalue: Print only the values of the FITS keywords given to this
option in separate columns. This option can take multiple keyword
names and many FITS files. Thus generating a table of keyword values
(with one row per file where the first column is the file name). Its
output can thus be written as a Table file or be piped to the Table
program for selecting a certain sub-set of your FITS files based on
key values, or sorting them for example. This was added after a
discussion with Alberto Madrigal.
MakeCatalog:
- Newly added measurement columns:
--upperlimitsb: upper-limit surface brightness for the given label
(object or clump). This is useful for measuring a dataset's
realistic surface brightness level for each labeled region by random
positioning of its footprint over undetected regions (not
extrapolated from the single-pixel noise level like the "surface
brightness limit").
NoiseChisel:
- Can now work on 3D datacubes. Since the configuration parameters are
different from images, it is recommended to manually set the 3D
configuration (the '...' can be the input image and options):
astnoisechisel --config=/usr/local/etc/astnoisechisel-3d.conf ...
Alternatively, you can set an 'astnoisechisel-3d' alias like below and
always easily run 'astnoisechisel-3d' on cubes.
alias astnoisechisel-3d="astnoisechisel --config=/usr/local/etc/astnoisechisel-3d.conf"
Segment:
- Can now work on 3D datacubes. Similar to NoiseChisel, it requires a
separate set of default configurations, so please see the note under
NoiseChisel above.
Table:
- When given a value of '_all', the '--noblank' option (that will remove
all rows with a blank value in the given columns) will check all
columns of the final output table. This is handy when you want a
"clean" (no NaN values in any column) table, but the table has many
columns. Until now, '--noblank' needed the name/number of each column
to "clean".
--rowlimit: new option to specify the positional interval of rows to
show. Until now, the '--head' or '--tail' options would just allow
seeing the first or last few rows. You can use this new option to view
a contiguous set of rows in the middle of the table.
--rowrandom: Make a random selection of the rows. This option is useful
when you have a large table and just want to see a random sub-set of
the rows. It takes an integer, selects that many rows from the input
randomly.
--polygon: Similar to same option in Crop (mentioned above).
- New column arithmetic operators:
- 'set-AAA' operator (which allows storing the popped operand into a
named variable for easy usage in complex operations) is also usable
in Table's column arithmetic. Until now this operator was only
available in the Arithmetic program (for operation on images).
- 'date-to-sec' Convert FITS date format ('YYYY-MM-DDThh:mm:ss') into
seconds from the Unix epoch (1970-01-01,00:00:00 UTC). This can be
very useful in combination with the new '--keyvalue' option of the
Fits program to operate on FITS dates (for example sort your FITS
images based on observation time).
Query:
- The Galactic extinction calculator of the NASA/IPAC Extragalactic
Database (NED) is now available for any coordinate with a command like
below. For more, see the manual (the description of the 'extinction'
dataset of NED in the "Available datasets" section).
astquery ned --dataset=extinction --center=49.9507,41.5116
This feature was suggested by Ignacio Trujillo and implemented with
the help of Joseph Mazzarella.
Library:
- New arithmetic operator macros (for the 'gal_arithmetic' function):
- GAL_ARITHMETIC_OP_SIN: sine (input in deg).
- GAL_ARITHMETIC_OP_COS: cosine (input in deg).
- GAL_ARITHMETIC_OP_TAN: tangent (input in deg).
- GAL_ARITHMETIC_OP_ASIN: Inverse sine (output in deg).
- GAL_ARITHMETIC_OP_ACOS: Inverse cosine (output in deg).
- GAL_ARITHMETIC_OP_ATAN: Inverse tangent (output in deg)
- GAL_ARITHMETIC_OP_ATAN2: Inverse tangent (with two inputs, out deg).
- GAL_ARITHMETIC_OP_SINH: Hyperbolic sine.
- GAL_ARITHMETIC_OP_COSH: Hyperbolic cosine.
- GAL_ARITHMETIC_OP_TANH: Hyperbolic tangent.
- GAL_ARITHMETIC_OP_ASINH: Inverse hyperbolic sine.
- GAL_ARITHMETIC_OP_ACOSH: Inverse hyperbolic cosine.
- GAL_ARITHMETIC_OP_ATANH: Inverse hyperbolic tangent.
- GAL_ARITHMETIC_OP_MKNOISE_SIGMA: Add fixed-sigma noise.
- GAL_ARITHMETIC_OP_MKNOISE_POISSON: Add Poisson noise.
- GAL_ARITHMETIC_OP_COUNTS_TO_JY: Convert counts to Janskys.
- New arithmetic flag macros:
- GAL_ARITHMETIC_FLAG_ENVSEED: read random number generator seed from env.
- GAL_ARITHMETIC_FLAG_QUIET: don't print any warnings by some operators.
- WCS coordinate system macros:
- GAL_WCS_COORDSYS_EQB1950: 1950.0 (Besselian-year) equatorial coords.
- GAL_WCS_COORDSYS_EQJ2000: 2000.0 (Julian-year) equatorial coords.
- GAL_WCS_COORDSYS_ECB1950: 1950.0 (Besselian-year) ecliptic coords.
- GAL_WCS_COORDSYS_ECJ2000: 2000.0 (Julian-year) ecliptic coords.
- GAL_WCS_COORDSYS_GALACTIC: Galactic coordinates.
- GAL_WCS_COORDSYS_SUPERGALACTIC: Supergalactic coordinates.
- gal_array_file_recognized: For FITS images, check contents also.
- gal_ds9_reg_read_polygon: Parse the polygon from an SAO DS9 region file.
- gal_fits_file_recognized: Check the file contents when suffix isn't FITS.
- gal_units_counts_to_mag: Convert counts to magnitudes.
- gal_units_mag_to_counts: Convert magnitudes to counts.
- gal_units_counts_to_jy: Convert counts to Janskys.
- gal_wcs_coordsys_from_string: WCS coordinate system from string.
- gal_wcs_coordsys_identify: Parse WCS struct to find coordinate system.
- gal_wcs_coordsys_convert: Convert the coordinate system of the WCS.
** Removed features
** Changed features
astscript-sort-by-night:
- Thanks to the new features in the Fits and Table programs (described
above), the efficiency of this script has improved dramatically (from
19 seconds to 0.42 seconds for about 650 FITS files used in the
test!).
- The default end to a "night" is set to 11:00a.m. Until now it was
9:00a.m. But in some cases, calibration images may be taken after
that. So to be safer in general it was incremented by 2 hours.
MakeCatalog:
- Surface brightness limit (SBL) calculations are now written as
standard FITS keywords in the output catalog/table. Until now, they
were simply stored as 'COMMENT' keywords with no name so it was hard
to parse them automatically. From this version, the following keywords
are also written into the output table(s), see the "MakeCatalog
output" section of the book for more: 'SBLSTD', 'SBLNSIG', 'SBLMAGPX',
'SBLAREA', 'SBLMAG'.
- Upper-limit (UP) settings are also written into the output tables as
keywords (like surface brightness limit numbers above): 'UPNSIGMA',
'UPNUMBER', 'UPRNGNAM', 'UPRNGSEE', 'UPSCMLTP', 'UPSCTOL'.
Library:
- GAL_ARITHMETIC_FLAG_FREE new name for GAL_ARITHMETIC_FREE.
- GAL_ARITHMETIC_FLAG_NUMOK new name for GAL_ARITHMETIC_NUMOK.
- GAL_ARITHMETIC_FLAG_INPLACE new name for GAL_ARITHMETIC_INPLACE.
- GAL_ARITHMETIC_FLAGS_BASIC new name for GAL_ARITHMETIC_FLAGS_ALL.
- gal_fits_key_write_wcsstr: also takes WCS structure as argument.
- gal_fits_key_read_from_ptr: providing a numerical datatype for the
desired keyword's value is no longer mandatory. When not given, the
smallest numeric datatype that can keep the value will be found and
used.
- gal_wcs_read: allows specifying the linear matrix of the WCS.
- gal_wcs_read_fitsptr: allows specifying the linear matrix of the WCS.
** Bugs fixed
bug #60082: Arithmetic library crash for integer operators like modulo
bug #60121: Arithmetic segfault when multi-operand output given to set-
bug #60368: CosmicCalculator fails --setdirconf when redshift isn't given,
reported by Sepideh Eskandarlou
bug #60483: No warning when file given to '--config' doesn't exist,
reported by Sepideh Eskandarlou
bug #60484: Match crashes when called with --coord and --ccol2 (together)
bug #60603: Table crashes with an empty input and --range,
reported by Sepideh Eskandarlou.
bug #60619: Crop crashes with differing image and WCS dimension
bug #60634: Crop's '--hstartwcs' and '--hendwcs' wrong with comment keys
bug #60644: MakeCatalog's Brightness error over-estimated in images with
noise STD less than 1. This was not an issue for S/N or
Magnitude error on the same image (already corrected there).
* Noteworthy changes in release 0.14 (library 12.0.0) (2021-01-25) [stable]
** New features
Book:
- Tutorial on "Detecting large extended targets" improved with better
NoiseChisel configuration, and more clear description.
- New sub-section on "Memory management" in the "Common program
behavior" chapter. It fully describes how to optimally deal with large
datasets that may exceed your system's RAM.
- Examples and better description added to many operators in the
"Arithmetic operators" subsection of the Arithmetic program's section.
New program:
- Query ('astquery') is a new program to query to external datasets and
retrieve the resulting datasets from the command-line. It is possible
to get list of datasets or column names from the databases. The basic
spatial query (finding objects in the vicinity of a certain point) can
be managed without knowing the query language of the database: with
basic options (like '--center' with '--radius' or '--width'), or an
input image that has WCS. Currently Query supports VizieR (containing
+20500 datasets, making it the largest database in astronomy), NED, as
ESA's Gaia database and ASTRON. See the new "Query" section of the
book (under the "Data containers" chapter) for more.
All programs:
- Plain text table inputs can have floating point columns that are in
sexagesimal format of '_h_m_s' or '_d_m_s' (where '_' is a
number). These are the classical format to respectively represent
Right Ascension (RA) and Declination (Dec). They will be directly read
as a single floating point number (in units of degrees) into
memory. Therefore the same column of a plain-text table, can be
degrees in some rows and sexagesimal in others. Besides large tables,
with this feature, conversion to sexagesimal coordinates to degrees
becomes very easy, for example:
echo "7h34m35.5498 31d53m14.352s" | asttable
Recall that the inverse can also be done with the more general column
arithmetic:
echo "113.64812416667 31.88732" \
| asttable -c'arith $1 degree-to-ra $2 degree-to-dec'
- If input is a HEALpix grid (1D table column that represents the 2D
spherical representation of datasets), the programs will print a
warning, suggesting to use the 'HPXcvt' utility of WCSLIB.
Arithmetic:
- New operators:
- 'interpolate-maxofregion': interpolate connected blank regions with
the maximum value that is immediately touching it. This can be used
to fill the blank centers of saturated stars for example.
- 'interpolate-minofregion': similar to 'interpolate-maxofregion', but
for the minimum.
- 'makenew': new operator to create an empty (zero-valued) new dataset
with given dimension and size (given as operands).
Crop:
--primaryimghdu: Write the final cropped image into the primary (or
0-th) extension of the output FITS file, so the output only has
one extension.
Fits:
- New '--skycoverage' option will report the area of the input image in
RA/Dec, both in units of center/width, and box minimum/maximum
format. This is paritcularly useful in combination with the new
'astquery' option, to easily search the contents of external databases
within the image.
MakeCatalog:
--maximum: maximum value of labeled regions pixels (clump/object).
--areaarcsec2: area of labeled region (clump/object) in arcsec^2.
--surfacebrightness: the surface brightness of the labeled region.
--fwhm: observed FWHM in pixels (non-parametric), along the major axis.
--halfmaxarea: number of pixels with a value larger than half the maximum.
--halfmaxradius: radius of region that is larger than half the maximum.
--halfmaxsum: sum of pixels with a value larger than half the maximum.
--halfmaxsb: surf. brightness within half of the maximum.
--fracmax: fractions to use in '--fracmaxarea1' or '--fracmaxarea2'.
--fracmaxsum1: sum of pixels brighter than first given fraction of max.
--fracmaxsum2: sum of pixels brighter than second given fraction of max.
--fracmaxarea1: number of pixels brighter than first given fraction of max.
--fracmaxarea2: number of pixels brighter than second given fraction of max.
--fracmaxradius1: radius derived from '--fracmaxarea1'.
--fracmaxradius2: radius derived from '--fracmaxarea2'.
--halfsumsb: Surface brightness within area reported by '--halfsumarea'.
--halfsumarea: area containing half of the summed object or clump values.
--halfsumradius: radius derived from '--halfsumarea', underestimates r_e.
--areaminv: the number of pixels that are equal to the minimum value.
--areamaxv: the number of pixels that are equal to the maximum value.
- New columns to return the position of pixel with minimum or maximum
value: '--minvx', '--maxvx', '--minvy', '--maxvy', '--minvz',
'--maxvz'.
MakeNoise:
--bgisbrightness: new option to say that the value of '--background'
(used to simulate Poisson noise) should be interpreted as brightness,
not magnitude.
MakeProfiles:
- It is now possible to make any custom radial profile with the 'custom'
profile (with code '8'). A table should be given to the new
'--customtable' option which will define each radial interval and the
value to use for that radial interval. See the description of
'--customtable' for more.
Statistics:
- 2D histograms can now be built as a FITS image with a linear WCS that
contains axis values and box size. This allows using the power of FITS
viewers for plotting/inspecting distributions of two columns in a
table relative to each other (for example color-magnitude plots). You
can also convert these 2D histogram images to PDF or JPEG with
Gnuastro's ConvertType to directly use in your papers/reports (see the
newly added "2D histogram as an image" section of the book for more).
Table:
- New '--noblank' option will remove all rows in output table that have
at least one blank value in the specified columns. For example if
'table.fits' has blank values (NaN in floating point types) in the
'magnitude' and 'sn' columns, with '--noblank=magnitude,sn', you can
be sure that all rows with blank values in these columns have been
removed.
- New trigonometric operators for column-arithmetic (inputs in units of
degrees): 'sin', 'cos' and 'tan'. Their inverse trigonometric (outputs
in units of degrees) have also been added: 'asin', 'acos' and 'atan'.
The 'atan2' operator (inverse tangent that preserves the quadrant, see
its description in the book) is also now available.
- New hyperbolic operators for column-arithmetic: 'sinh', 'cosh' and
'tanh'. Their inverse has also been added: 'asinh', 'acosh' and
'atanh'.
Library:
- GAL_ARITHMETIC_OP_MAKENEW: new 'makenew' operator.
- gal_binary_connected_adjacency_list: finding connected components
without a square adjacency matrix (which can consume major RAM).
- gal_blank_flag_remove: Remove all flagged elements in a dataset.
- gal_blank_remove_rows: remove all rows that have at least one blank.
- gal_dimension_dist_elliptical: Elliptical dist. of a point from center.
- gal_fits_hdu_is_healpix: Return 1 if HDU is a HEALpix grid.
- gal_fits_hdu_datasum: calculate DATASUM of given HDU in given FITS file.
- gal_fits_hdu_datasum_ptr: calculate DATASUM of opened FITS file pointer.
- gal_fits_key_list_comment_add: add a COMMENT keyword to the keyword list.
- gal_fits_key_list_comment_add_end: add a COMMENT keyword to end of list.
- gal_pointer_allocate_ram_or_mmap: allocate in RAM or memory-mapped file.
- gal_pointer_mmap_free: "free" (actually delete) the memory-mapped file.
- gal_wcs_create: create WCSLIB-compatible WCS from raw values.
- gal_wcs_coverage: Return the sky coverage of given image HDU.
- gal_wcs_dimension_name: return the name of the requested WCS dim.
** Removed features
** Changed features
All programs:
- Memory management: Until now, an internal array was only allocated in
the RAM when its size was smaller (in bytes) than the value given to
the '--minmapsize' option. But this was annoying/buggy when the system
has enough RAM to keep large files. From this version, all Gnuastro
programs will first attempt to write the array in RAM, only when it
fails (there is no more RAM left), will they use a memory-mapped file
(which can dramatically slow down the program). Please see the newly
added "Memory management" section of the book for a complete
explanation of Gnuastro's new memory management strategy.
- When an array needs to be memory-mapped (read into a file on HDD/SSD,
not RAM, usually due to RAM limitations), it is written in a
'gnuastro_mmap' directory of the running directory, not the hidden
'.gnuastro_mmap' directory. Since the files in this directory are
usually large, it is better that the user sees them by default (in
case the program crashes and they aren't deleted).
--interpnumngb: the default value has been increased to 15 (from 9). The
reason for this is that we now have a more robust outlier removal
algorithm (see description under "NoiseChisel & Statistics").
Crop:
- When cropping a single image in WCS mode, there is no longer any
limitation on the WCS. Until now for all WCS mode crops, it was
necessary for the WCS to be aligned to the celestial coordinates. But
from this version, this is only necessary when cropping from many
files (and stitching them together where necessary). For WCS-mode
crops of a single image, any WCS that is recognized by WCSLIB is fine.
Fits:
- The '--pixelscale' option also prints the pixel area (for 2D inputs,
or 2D slices of 3D inputs) and the voxel volume (for 3D inputs). Until
now, it would only print the pixel scale along each dimension.
- When printing FITS file HDU information (no options given), a new
"Comments" column may be printed for each HDU in the end of the line.
It will be printed if special situations are found (for example a 2D
HEALPix grid, that is usually stored as a 1D array/column).
NoiseChisel & Statistics:
- New algorithm used to reject outlying tiles. In NoiseChisel this is
done when estimating the quantile threshold, the pseudo-detection
threshold and the final Sky value. In Statistics, its just the Sky
value. Unlike the previous method that used the global distribution of
tile values to identify outliers, the new algorithm uses a relative
measure. See the book for more. Since outliers are now rejected more
robustly, the default value of '--meanmedqdiff' has been increased to
0.01 (was 0.005) and '--outliersigma' has been decreased to '5' (was
10). Also 'smoothwidth' has been increased from '3' to '5' to have
smoother tessellation values.
Statistics:
- The '--histogram2d' now takes a string argument: either 'image' or
'table'. For the old behavior please use '--histogram2d=table'. See
the new features above for the 'image' mode.
Table:
- Column arithmetic operators 'degree-to-ra' and 'degree-to-dec' will
return the sexagesimal format of '_h_m_s' and '_d_m_s'
respectively. Until this version, they would both use colons as
delimiters ('_:_:_').
Library:
- gal_pointer_mmap_allocate: new name for 'gal_pointer_allocate_mmap'.
- gal_threads_dist_in_threads: now accounts for billions of threads,
thus includes memory management options.
- gal_threads_spin_off: now accounts for memory management.
- gal_units_degree_to_ra: new 'usecolon' argument to optionally format
output string with colons as delimiters ('_:_:_'). When this option is
zero, the string will be in the '_h_m_s' format.
- gal_units_degree_to_dec: similar to 'gal_units_degree_to_ra', but when
'usecolon' is zero, the string will be in the '_d_m_s' format.
** Bugs fixed
bug #59017: Segment's object IDs are not thread-safe (i.e., reproducible).
bug #59105: Column arithmetic operator degree-to-ra, returning to dec.
bug #59136: Makeprofiles with --replace is not thread-safe.
bug #59155: Match cannot find the proper row when coordinates have NaN.
bug #59371: MakeCatalog crash with clumps on non-contiguous object labels.
bug #59400: CosmicCalculator fails --printparams when redshift isn't given.
bug #59459: Unclear WCS when both PC and CD exist in input, but conflict.
bug #59625: MakeProfiles uses last --kernel, if it is called more than once.
bug #59700: Segment's excessive RAM usage when many clumps over a detection.
bug #59765: MakeCatalog crash when to-be-subtracted Sky is a single-element
per tile tessellation (e.g., NoiseChisel's '--oneelempertile').
* Noteworthy changes in release 0.13 (library 11.0.0) (2020-09-06) [stable]
** New features
All programs:
- When reading plain-text tables, the blank value for numeric columns
can be any string (specified in the special comment-line format
described in the "Gnuastro text table format" of the manual). Until
now, it had to be a number in the same type.
Arithmetic:
- New operators:
- interpolate-minngb: Fill blanks with minimum of nearest neighbors.
- interpolate-maxngb: Fill blanks with maximum of nearest neighbors.
This can be useful to fill the blank values of saturated stars
for example.
- To force integers to floats, you can also put a '.' or '.0' after
them. Until now, it was only possibly by putting an 'f' after
them. Hence while '5' will be read as an integer, '5.', '5.0' or '5f'
will be read as floating point. This also applies to column arithmetic
in Table.
ConvertType:
- New colormap: 'sls-inverse' is the inverse of the SLS color, good for
printing because of a white background.
CosmicCalculator:
--velocity: Velocity (in km/s) to use instead of input redshift.
--usedvelocity: Print the velocity (in km/s) at input redshift.
--listlinesatz: Print the wavelength of all pre-defined spectral lines
at given redshift as a simple table with the line names. This is very
convenient and can be used in conjunction with '--obsline' for example
to print the observed wavelength of all lines when Lyman-alpha is at
4000 Angstroms with this simple command:
astcosmiccal --obsline=lyalpha,4000 --listlinesatz
FITS:
- New '--pixelscale' option will return the size of pixels in each
dimension in the "world coordinates".
- New '--wcsdistortion' option allows conversion between the various WCS
distortions. For example if you have a FITS image with the TPV-based
WCS distortion, and you would like to convert it to a SIP-based
distortion, you can simply run 'astfits --wcsdistortion=SIP' on the
file. The inverse conversion is also supported (from SIP to TPV).
Statistics
- New feature to create a 2D-histogram using two input columns (useful
when you have lots of points that are too dense and may hide important
features). This mode can be activated with the new '--histogram2d'
option. The binning of the first (X axis) column is specified with the
same 1D histogram options. The second column's binning is configured
with the following options:
--numbins2: Number of bins along the second column.
--greaterequal2: Only second column points that are larger than this.
--lessthan2; Only second column points that are less than this.
--onebinstart2: Make sure one bin starts at the value given here.
Table:
- New '--catcolumns' to specify which columns to concatenate (or append)
to the output. You can specify the file name containing the columns to
append with the '--catcolumnfile' option and '--catcolumnhdu' (see
changed features because until now they had different names).
- New '--catcolumnsrawname' will leave the name of concatenated
(appended) columns unchanged. By default the names of the appended
columns will be appended with a '-N' (where 'N' is a counter for the
file that is used to append columns). The default behavior is to avoid
multiple columns having the same name.
- New '--colmetadata' option to add/update column metadata (name, units
or comments) just before writing the output. This is a very useful
feature in combination with column arithmetic or column concatenation
because it will allow you to update the new column metadata in the
same command. See the manual for more.
Library:
- Spectral lines library: SiIII, OIII, CIV, NV and rest of Lyman series.
- GAL_CONFIG_HAVE_WCSLIB_DIS_H: if the host's WCSLIB supports distortions.
- GAL_CONFIG_HAVE_WCSLIB_MJDREF: if the host's WCSLIB reads MJDREF keyword.
- gal_cosmology_velocity_from_z: Calculate velocity from redshift.
- gal_cosmology_z_from_velocity: Calculate redshift from velocity.
- gal_data_array_ptr_calloc: Allocate array of pointers to gal_data_t
- gal_data_array_ptr_free: Free all the datasets within the array and itself.
- gal_fits_key_list_title_add: Add a title key word to the list.
- gal_fits_key_list_title_add_end: Add a title key word to the list's end.
- GAL_INTERPOLATE_NEIGHBORS_METRIC_RADIAL: Radial metric for interpolation.
- GAL_INTERPOLATE_NEIGHBORS_METRIC_MANHATTAN: Mahattan distance.
- GAL_INTERPOLATE_NEIGHBORS_METRIC_INVALID: For error-handling/completeness.
- GAL_INTERPOLATE_NEIGHBORS_FUNC_MIN: Use minimum for interpolation.
- GAL_INTERPOLATE_NEIGHBORS_FUNC_MAX: Use maximum for interpolation.
- GAL_INTERPOLATE_NEIGHBORS_FUNC_MEDIAN: Use median for interpolation.
- GAL_INTERPOLATE_NEIGHBORS_FUNC_INVALID: for error-handling/completeness.
- gal_kdtree_create: Create a k-d tree for optimal spatial searching.
- gal_kdtree_nearest_neighbour: Find the nearest neighbor using a k-d tree.
- gal_statistics_histogram2d: Generate 2D histogram from two columns.
- GAL_WCS_FLTERROR: Limit to identify a floating point error for WCS.
- gal_wcs_write: Write the given WCS into a FITS extension with no data.
- gal_wcs_clean_errors: clean major WCS components from errors specified
by the FITS keyword 'CRDER', or floating point errors.
- gal_wcs_distortion_from_string: Return distortion string/name from ID.
- gal_wcs_distortion_to_string: Return distortion ID from string/name.
- gal_wcs_distortion_identify: Identify the distortion of given WCS.
- gal_wcs_distortion_convert: Convert between various WCS distortions.
** Removed features
** Changed features
Arithmetic:
- The 'pow' operator can also accept integer inputs. This also applies
to column arithmetic in Table.
MakeProfiles:
- The status of every created profile (along with the number of
remaining profiles) is no longer printed when there are more than 50
profiles. This is done because printing itself can slow down the
program an in a general/automated script this info is redundant.
Table:
--catcolumnfile ('-L') is new name for '--catcolumn' ('-C').
--catcolumnhdu is new name for '--catcolhdu' (short option name hasn't
changed).
Library:
- gal_fits_key_list_add: new 'ufree' argument to optionally free units.
- gal_fits_key_list_add_end: similar to 'gal_fits_key_list_add'.
- gal_interpolate_neighbors: new name for gal_interpolate_close_neighbors.
- gal_statistics_outlier_bydistance: new name for the old
'gal_statistics_outlier_positive'. It can now use the same algorithm
for negative outliers with a new argument.
- gal_txt_write: Now accepts a new argument for keyword lists.
- gal_type_string_to_number: Numbers ending in '.' or '.0' will be
parsed as floating point. Until now, it would only parse numbers as
floating point if they had non-zero decimals.
** Bugs fixed
bug #58434: MakeCatalog crash when ordering is required and no usable pixels.
bug #58455: Timezone is not portable and uses flag instead of seconds.
bug #58696: Warp with --centeroncorner --scale making wrong size.
bug #58774: Warp' s output on a cube is a 2D image or wrong size.
bug #58809: NoiseChisel not removing negative outlier tiles.
bug #58833: Segment crashes when detection map has blank pixels.
bug #58835: Floating point errors when comparing pixel scale in Crop.
bug #58898: Plain text string columns touching next, clear first character.
bug #58901: Blank values for non-standard integer types in FITS tables.
bug #58974: WCS conversion not reasonable on processed TPV data.
bug #59019: FITS Table crash when TFORM comes before TNULL.
* Noteworthy changes in release 0.12 (library 10.0.0) (2020-05-20) [stable]
** New features
Arithmetic:
- New 'quantile' operator for coadding datasets.
- New 'size' operator to report length of dataset in requested dimension.
- When '--wcsfile' is given the value 'none', output will not have any WCS.
CosmicCalculator:
--listlines: list the pre-defined spectral line wavelengths and names
(which you can use with the '--obsline' and '--lineatz' options). This
is convenient when you forget the specific name of the spectral line
used within Gnuastro.
Crop:
--polygon: can now also crop concave polygons (when atleast one inner
angle is more than 180 degrees). Concave polygons occur a lot in deep
astronomical imaging: in the shape of the deepest regions.
--polygonsort: Sort the given set of vertices to the '--polygon'
option. For a concave polyton, the sorting will be correct, but for a
convex polygon, there is no unique solution/sorting, so it may not be
what you expect, see the manual.
Fits:
--datasum: Calculate and print the given HDU's "datasum" to stdout.
--datetosec: Can also account for 'Z' in the end of the date-time
string. According to 'https://www.w3.org/TR/NOTE-datetime', a 'Z'
effectively means no time zone, or UTC time (which is the default in
FITS). It still doesn't account for time zone hours of the w3.org
standard.
MakeCatalog:
--sigmaclip: define sigma-clipping parameters for the new '--sigclip-*'
columns.
--forcereadstd: Read the standard deviation image even if not needed by
any columns. This is useful when you want the surface brightness
limit, but don't need any error-related columns.
New output columns:
--sigclip-number: Number of sigma-clipped pixels in object/clump.
--sigclip-median: Sigma-clipped median of pixels in object/clump.
--sigclip-mean: Sigma-clipped mean of pixels in object/clump.
--sigclip-std: Sigma-clipped standard deviation of pixels in object/clump.
Table:
--equal: Can now work on columns with string type also.
--notequal: Can now work on columns with string type also.
--polygon: Polygon to use in '--inpolygon' or '--outpolygon'.
--inpolygon: Select rows that are inside the polygon of '--polygon'.
--outpolygon: Select rows that are outside the polygon of '--polygon'.
--catcolumn: Concatenate tables by column (keeping number of rows fixed).
--catcolhdu: Specify the HDU/extension of the FITS files of --catcolumn.
- New operators in column arithmetic:
- 'ra-to-degree': Convert Right Ascension (HH:MM:SS) to degrees.
- 'dec-to-degree': Convert Declination (DD:MM:SS) to degrees.
- 'degree-to-ra': Convert degrees to Right Ascension (HH:MM:SS).
- 'degree-to-dec': Convert degrees to Declination (HH:MM:SS).
- 'distance-flat': Distance between two points, assuming flat space.
Library:
- GAL_SPECLINES_INVALID_MAX: Total number of spectral lines, plus 1.
- GAL_ARITHMETIC_OP_QUANTILE: operator for 'gal_arithmetic'.
- gal_txt_trim_space: trim white space before and after a string.
- gal_polygon_is_convex: identify if a polygon is convex or concave.
- gal_polygon_is_inside: if point is inside polygon (convex or concave).
- gal_polygon_is_counterclockwise: check if polygon is counter-clockwise.
- gal_polygon_to_counterclockwise: convert to counter-clockwise if it isn't.
- gal_polygon_vertices_sort: un-ordered vertices to concave/convext polygons.
- gal_units_extract_decimal: Extract numbers from strings like "A:B:C".
- gal_units_ra_to_degree: Convert RA (HH:MM:SS) to degrees.
- gal_units_dec_to_degree: Convert Dec (DD:MM:SS) to degrees.
- gal_units_degree_to_ra: Convert degrees to RA (DD:MM:SS).
- gal_units_degree_to_dec: Convert degrees to Dec (DD:MM:SS).
** Removed features
** Changed features
All programs and libraries:
--minmapsize: Gnuastro's programs no longer attempt to write
memory-mapped files under '.gnuastro'. They will only attempt to write
them under the '.gnuastro_mmap' directory. Until now, when an internal
array needed to be memory-mapped, Gnuastro's programs (through the
'pointer.h' library) would first try writing the mmap files in the
'.gnuastro' directory. When it failed it would attempt writing in the
'.gnuastro_mmap' directory. However, '.gnuastro' is also used to store
configuration files (which are hand-written and thus valuable). Mixing
the two types of source (configuration files) and automatically
generated (memory-mapped) files is very problematic.
- FITS ASCII tables: When a column has a floating point type, but its
ASCII string can't be parsed as a number, it will be read as a
NaN. Until now, the corresponding program/library would abort,
printing the problematic string and its location.
Crop:
--polygon: by default it will no longer attempt to sort the polygon
vertices, sorting can be requested with the new '--polygonsort' option.
--polygonout: is the new name for '--outpolygon'. Having 'polygon' at
the start of the option name, makes it easier to find in the help list
and also to understand generally.
MakeCatalog:
- Until now, if no standard deviation image was requested, MakeCatalog
wouldn't include any surface brightness limit metadata in its
output. Now, those two lines are filled, but with a notice on the
cause (that there is no standard deviation image), and suggesting
solutions.
NoiseChisel:
- Until now, when NoiseChisel didn't detect any pixels, it just printed
a message and wouldn't not make any output file. This was very
inconvenient in general scripts. From now on, in this scenario, an
output file will be created and the detection map will only have a
value of zero. As a result, the Sky and Sky standard deviation
extensions will be measured over all the tiles.
Table:
- In Column arithmetic, when columns must be specified by their number,
that number should be distinguished with a '$' before it (for example
'$1' means the first column). Until now, this character was 'c', but
the new identifying character is very similar to AWK, allowing easier
adoption and is also more clear. It is just important to put the total
'arith' string within single quotes, not double quotes.
- Operators:
- 'distance-on-sphere': New name for old `angular-distance' operator.
Library:
- gal_polygon_is_inside_convex: new name for 'gal_polygon_pin'.
** Bugs fixed
bug #57300: MakeCatalog memory crash when input dataset has units.
bug #57301: MakeCatalog using river sum instead of mean times by clump area.
bug #57921: Arithmetic's interpolation operator not reading metric.
bug #57989: Warp not accounting for translation in pixel grid.
bug #57995: Fits lib's date to second function affected by host's timezone.
bug #58315: Some NaNs with sigma-clip operators in Arithmetic and one input.
bug #58371: Table crashes with a commented newline in the columns.
* Noteworthy changes in release 0.11 (library 9.0.0) (2019-11-25) [stable]
** New features
Book:
- The "General program usage tutorial" now has a section on how to write
scripts effectively to automate your analysis.
Arithmetic:
- The new 'add-dimension' operator will stack the popped operands in a
higher-dimensional dataset. For example to build a 3D cube from
individual 2D images/slices.
--onedonstdout: when the output is one-dimensional, print the values on
the standard output, not into a file.
BuildProgram:
- Will use common environment variables like LDFLAGS, CPPFLAGS and CC to
help in customizing the build of your program.
--cc: custom C compiler to use. Until now, 'gcc' was hard-coded into the
source and there was no way to choose a custom C compiler.
--noenv: With this option, no environment variables will be read.
ConvertType:
- New 'viridis' colormap (value for the '--colormap' option). This is
the default colormap of Python's Matplotlib, and is available in many
other plotting tools like LaTeX's PGFPlots.
Convolve:
- Spatial domain convolution now possible on 3D data cubes (with a 3D
kernel).
CosmicCalculator:
--lineatz: return the observed wavelength of a line if it was emitted at
the redshift given to CosmicCalculator. You can either use known line
names, or directly give a number as any emitted line's wavelength.
MakeCatalog:
- Catalogs from 3D inputs now available, with the following new options,
see book for more.
--spectrum: label's spectrum (across the third dimension).
--z: Flux-weighted position in 3rd dimension.
--geoz: Geometric center in third FITS axis.
--minz: Minimum third FITS axis position.
--maxz: Maximum third FITS axis position.
--clumpsz: Flux weighted center of all clumps in object in 3rd dim.
--clumpsgeoz: Geometric center of all clumps in object in 3rd dim.
--w3: Flux weighted center in third WCS axis.
--geow3: Geometric center in third WCS axis.
--clumpsw3: Flux wheighted center of all clumps in 3rd dim.
--clumpsgeow3: Geometric center of all clumps in 3rd dim.
--areaxy: Projected area in first two dimensions.
--geoareaxy: Projected geoarea in first two dimensions.
--inbetweenints: output will contain one row for all integers between 1
and the largest label in the input (irrespective of their existance in
the input image). This was the default/only behavior of MakeCatalog
until now. However, there are situations where the labeled input image
integers may not be contiguous. For example if the input's only
labeled pixel values are 11 and 13 from this release MakeCatalog's
output will only have two rows. If you want the old behavior (of one
row per integer, even if its not in the image), you can use this
option.
MakeProfiles:
- Can produce mock ellipsoids in a datacube (using X-Z-X Euler angles
for 3D orientation), the following options have been added, see the
book for more details.
--p2col: Second Euler angle (X-Z-X order).
--p3col: Third Euler angle (X-Z-X order).
--q2col: Axis ratio (major/dim3 in 3D).
- The '--kernel' option can build 3D kernels, see the description of
this option in the book for examples and details on how to run it.
Match:
- Matching of catalogs now possible using 3 coordinates (on catalogs
generated from 3D data cubes), see book for more.
NoiseChisel:
- arXiv:1909.11230 added in papers to cite (with the '--cite' option):
this paper describes the major changes made to NoiseChisel in the last
10 stable releases since the 2015 paper, most importantly how Segment
has been separated and the new growth strategy. It is therefore
necessary to cite it along with the initial 2015 paper when using
NoiseChisel.
Segment:
- arXiv:1909.11230 added in papers to cite (with the '--cite' option):
this paper describes why Segment has been separated from NoiseChisel
and some important updates to it compared to the 2015 paper, it is
therefore necessary to cite it along with that paper when using
Segment.
Statistics:
--contour: compute a contour plot which can be directly fed into the
PGFPlots package of LaTeX for plotting the contours. Support for more
formats will be added based on the need/request.
Table:
--equal: Output only rows that have a value equal to the given value in
the given column. For example '--equal=ID,2,4,5' will select only rows
that have a value of 2, 4 and 5 in the 'ID' column.
--notequal: Output only rows that have a different value compared to the
values given to this option in the given column.
- Column Arithmetic operators:
- 'angular-distance': a new operator to easily find the angular
distance (along a great circle) between points in various table
columns, or the distances of all the points in the table rows with a
fixed point. See the book for examples and better explanation.
Library:
- gal_binary_connected_indexs: store indexs of connected components.
- gal_blank_remove_realloc: Remove blanks and shrink allocated space.
- gal_box_bound_ellipsoid_extent: Extent of 3D ellipsoid.
- gal_box_bound_ellipsoid: Bounding box for a 3D ellipsoid.
- gal_statistics_unique: Return unique (non-blank) elements of the input.
** Removed features
** Changed features
** Bugs fixed
bug #56736: CosmicCalculator crash when a single value given to --obsline.
bug #56747: ConvertType's SLS colormap has black pixels which should be orange.
bug #56754: Wrong sigma clipping output when many values are equal.
bug #56999: Compilation error on macOS 10.9 not recognizing AT_FDCWD.
bug #57057: BuildProgram not using environment LDFLAGS or CPPFLAGS.
bug #57101: Crop segmentation fault when no overlap exists in image-mode.
bug #57164: MakeCatalog crashes when a label isn't in the dataset.
bug #57180: MakeCatalog reporting infinity S/N when --instd isn't an image.
bug #57200: Generated pkgconfig must request wcslib, not wcs.
bug #57293: NaN value for brightness-related columns when values have NaN.
* Noteworthy changes in release 0.10 (library 8.0.0) (2019-08-03) [stable]
** New features
Installation:
- With the the following options at configure time, its possible to
build Gnuastro without the optional libraries (even if they are
present on the host system): '--without-libjpeg', '--without-libtiff',
'--without-libgit2'.
All programs:
- When an array is memory-mapped to non-volatile space (like the
HDD/SSD), a warning/message is printed that shows the file name and
its size. Later, when its deleted, a warning/message is also printed,
informing you that it has been deleted. These warnings can be very
useful when you actually have enough RAM, but forget to increase the
'--minmapsize' value (therefore significantly slowing down the
program). When you don't have enough RAM, but don't want to be annoyed
with the warnings, you can use the new '--quietmmap' option to disable
them.
Arithmetic:
- 'unique' operator removes all duplicate (and blank) elements from the
dataset and returns a single-dimension output, containing only the
unique values in the dataset.
Crop:
- Can also crop 3D datasets (data cubes). A 3D crop has the same syntax
as the old 2D mode, only when the dataset is 3D, three coordinates
(values, ranges or catalog-columns) should be given to the relevant
option. Just note that '--polygon' crops are still not supported in
3D.
CosmicCalculator:
--obsline: alternative way to set the used redshift. With this option
instead of explicitly giving the redshift, you can give a rest-frame
and observed wavelength and the redshift to use will be calculated
internally. For many lines, it is possible to give the line name
instead of its rest-frame wavelength. For example
'--obsline=lyalpha,6000' will use the redshift where the Lyman-alpha
line has been shifted to 6000 Angstroms.
--usedredshift: Print the used redshift as a "Specific calculation" (in
line with other single-valued calculations).
Fits:
--primaryimghdu: Copy/cut the given image HDU to the zero-th/first
HDU of the output file that doesn't yet exist.
Statistics:
--sigclip-number, --sigclip-median, --sigclip-mean, --sigclip-std: Do
sigma-clipping and only print the desired value as a single-value
measurement. Until now sigma-clipping results included a lot of
visually useful information, which also made automatic usage of
results hard. These options fix this issue. Please see the example in
the book under '--sigclip-median' for a nice use case.
Table:
- Column arithmetic. It is now possible to apply many operations on the
input table columns before printing them in the output. Similar to
Arithmetic, but on table columns. The operators and notation is just
like the Arithmetic program. See the "Column Arithmetic" section of
the book for a detailed discussion and several examples.
- WCS to Image coordinate conversion with 'wcstoimg' and 'imgtowcs'. For
example if the input catalog has at least an 'ID' column and two 'RA'
and 'DEC' columns, the set of options below will produce 5 columns
where the last two columns are the image coordinates for each row
based on the WCS in 'a.fits':
'-cID,RA,DEC -c"arith RA DEC wcstoimg" --wcsfile=a.fits'
--head: Only output the given number of rows from the top of columns.
--tail: Only output the given number of rows from the bottom of columns.
Library:
- New 'speclines.h' library functions and macros related to spectral
lines. It has many macros with line wavelengths, and several functions
for using them in combination with their names.
- list.h: Functions to return the last element in linked lists. For
example 'gal_list_sizet_last' or 'gal_list_data_last'.
- gal_arithmetic_operator_string: Return operator string from code.
- gal_arithmetic_set_operator: Return operator code from string.
- gal_blank_initialize_array: Initialize an array with blank values.
- gal_dimension_remove_extra: Remove extra (length 1) dimensions.
- gal_list_data_to_array_ptr: Make an array of pointers from the list.
- gal_fits_img_info_dim: Only return the size information of a dataset.
- GAL_TYPE_INT: Corresponding to respective width based on system.
- GAL_TYPE_UINT: Corresponding to respective width based on system.
- GAL_BLANK_INT: Blank value for 'int' (can be 16-bit or 32-bit).
- GAL_BLANK_UINT: Blank value for unsigned 'int' (can be 16-bit or 32-bit).
** Removed features
** Changed features
Installation:
- Better './configure' tests (using Gnulib's 'AC_LIB_HAVE_LINKFLAGS') to
avoid some crashes during 'make' when the host had multiple
conflicting versions of some dependencies (GSL in particular).
Arithmetic:
- The output of co-adding operators is no longer the same type as the
input in general. The output of the 'min' and 'max' operators are
still the same type as the input. However the 'number' and
'sigclip-number' operators will output an unsigned 32-bit integer type
and the rest ('sum', 'mean', 'std', 'median', 'sigclip-median',
'sigclip-mean' and 'sigclip-std') return 32-bit floating point
datasets
MakeCatalog:
- When a clumps catalog is requested, MakeCatalog will automatically
deduce the total number of clumps (at a small cost in
performance). Until now, it was mandatory for the clumps label dataset
to contain the total number of clumps in the 'NUMLABS' keyword.
Library:
- gal_statistics_outlier_flat_cfp: Improved implementation with new API.
- New 'quietmmap' argument added to the following functions (as the
argument following 'minmapsize'). For more, see the description above
of the new similarly named option to all programs: 'gal_array_read'
'gal_array_read_to_type', 'gal_array_read_one_ch',
'gal_array_read_one_ch_to_type', 'gal_data_alloc',
'gal_data_initialize', 'gal_fits_img_read',
'gal_fits_img_read_to_type', 'gal_fits_img_read_kernel',
'gal_fits_tab_read', 'gal_jpeg_read', 'gal_label_indexs',
'gal_list_data_add_alloc', 'gal_match_coordinates',
'gal_pointer_allocate_mmap', 'gal_table_read', 'gal_tiff_read' and
'gal_txt_image_read'
Book:
- The two larger tutorials ("General program usage tutorial", and
"Detecting large extended targets") have been broken into subsections
for easier readability.
- The "Hubble visually checks and classifies his catalog" tutorial has
been removed because it didn't come with a dataset, so it was hard for
people to use. Also, all its operations were already present in the
general tutorial.
** Bugs fixed
bug #56195: astscript-sort-by-night crashing because of AWK.
bug #56246: Single-valued measurement must abort with no value in Statistics.
bug #56256: Segmentation fault when reading plain text array/image.
bug #56257: ConvertType: Values not preserved when converting text to FITS.
bug #56299: CosmicCalculator fails at z=0.
bug #56324: Column metadata not usable when input is from pipe/stdin.
bug #56424: Warp crashes with empty string given to options.
bug #56480: Segfault in statistics library's histogram function.
bug #56641: MakeProfile's center position changes based on precision.
bug #56635: Update tutorial 3 with bug-fixed NoiseChisel.
bug #56662: Converting -R to -Wl,-R causes a crash in configure on macOS.
bug #56671: Bad sorting with asttable if nan is present.
bug #56709: Segment crash when input has blanks, but labels don't.
bug #56710: NoiseChisel sometimes not including blank values in output.
* Noteworthy changes in release 0.9 (library 7.0.0) (2019-04-17) [stable]
** New features
All programs:
--checkconfig: print the names and values given to options as they are
parsed on the command-line or in various configuration files (the
configuration file name is also printed). This can be very useful in
debugging (finding which configuration file is responsible for a given
option's value).
Arithmetic:
- The new 'tofile-' and 'tofilefree-' operators will save the top
operand into a file. They can be very handy in debugging/understanding
an Arithmetic command (especially as it gets complicated), or to
produce multiple files/extensions with a single call to Arithmetic.
- Four new operators have been added to allow coadding multiple datasets
into one using sigma-clipping: 'sigclip-number', 'sigclip-mean',
'sigclip-median', and 'sigclip-std'. These are very useful when
several inputs have strong outliers that affect the median, or the
mean is required.
- Multithreaded operation for the following operators that
combine/co-add multiple inputs into one output with same size: 'min',
'max', 'number', 'sum', 'mean', 'std', 'median', 'sigclip-number',
'sigclip-median', 'sigclip-mean', 'sigclip-std'.
--wcsfile and --wcshdu: these two options can be used to specify a
different file for reading the WCS of the output. This is useful when
the default (the WCS of the first dataset that is read) is not the
required one.
--interpmetric: new option that is necessary with the
'interpolate-medianngb' operator. For more, see the description of
this option in NoiseChisel.
Fits:
- Add "title" to group FITS keywords with '--write=/,"title name". This
"title" is composed of two keyword records/lines: a blank one (all
whitespace), followed by another starting with '/' and ending in any
string given to this option. This visually separates the keywords and
acts as a title. Classifying the keywords into contextually similar
groups greatly helps in visual inspection and is encouraged.
- Calculate and write 'CHECKSUM' and 'DATASUM' integrity keywords into
the specified header using '--write=checksum' (for both) or
'--write=datasum' (only for 'DATASUM').
--datetosec: Convert the FITS date format (old or new) to number of
seconds since since the Unix epoch time (1970-01-01,00:00:00). The
FITS date format (for example 'YYYY-MM-DDThh:mm:ss') is hard to use
for automatic processing (requires calendar peculiarities like number
of days in each month, or leap years and etc). With this option a
single integer is returned that can be used for example to sort FITS
files by date keywords without worrying about calendar peculiarities.
--verify: confirm if the 'DATASUM' and 'CHECKSUM' keyword values agree
with the specified HDU's content and/or data.
--copykeys: Copy several keyword records (in a given range) from one
FITS HDU/extension into another (possibly in another file).
--outhdu: The name/number of the output HDU (for '--copykeys').
Match:
- All the columns from one of the input catalogs can now be merged with
any of the columns of the second using the special '_all' name of
'--outcols'. For example the output of '--outcols=a_all,b5' will
contain all the columns from the first input and the 5th column of the
second input. This greatly simplifies the merging of different table
columns into one.
--coord: manually specify coordinates to match on the
command-line. Until now, if you only wanted to make check a specific
coordinate's matching with a catalog. It was necessary to make a
single-row catalog as a file and feed that into Match. With this
option, you can now specify the coordinates to match against another
catalog with the command-line.
NoiseChisel:
--interpmetric: Set the metric to use to identify the nearest neighbors
for tile interpolation (quantile threshold, initial Sky, and final
Sky). Until now only the manhattan/taxicab metric was available, which
is fast, but could cause 45-degree lines in the interpolation. From
this version, with this option, its also possible to use the radial
distance (which is now the default). Just note that if there are many
tiles over the image, the radial distance will be slower.
--snthresh: Manually set the signal-to-noise ratio of true
pseudo-detections. With this option, NoiseChisel will not attempt to
find pseudo-detections over the noisy regions of the dataset, but will
directly go onto applying the manually input value.
- Several new options have been added to remove NoiseChisel's dependence
on values that were hard-coded in its source and thus not modifiable
at run-time by the user. To allow full configurability, these steps
can also be configured by the user.
--pseudoconcomp: allows setting the connectivity (4 or 8, in a 2D
image) to define separate pseudo-detections. If its stronger,
pseudo-detections that are touching on the corner will be identified
as one.
--dopening: The number of openings to do after applying '--dthresh'.
--dopeningngb: The connectivity (4 or 8) to use for '--dopening'.
Statistics:
--interpmetric: Similar to NoiseChisel.
Table:
--range: Limit the output rows to those with a value within the given
numeric range with this format: '--range=COL,low,high'. This is very
useful when only certain rows of the input are required not the
output. The advantage over piping to AWK is that you can save the
output directly to FITS (preserving the metadata). See the book for
more.
--sort: Sort the output rows based on the value of the given column in
ascending order.
--descending: When called with '--sort', will arrange the output rows in
descending order.
Installed scripts:
With this release, Gnuastro also installs Bash scripts for common
higher-level usage of (possibly multiple) programs. These scripts have a
'astscript-*' name, to easily show up on the command-line as Gnuastro
executables with the other Gnuastro programs, but are identifiable from
them. They support options just like the programs (which can be listed
with '--help'). Please see the new "Installed scripts" chapter of the
book for more.
- astscript-sort-by-night: New Gnuastro executable, using Gnuastro's
Fits program to identify files with dates in the same night (possibly
spanning two calendar dates).
Library:
GAL_BLANK_LONG: new macro for the 'long' type blank value.
GAL_BLANK_ULONG: new macro for the 'unsigned long' type blank value.
gal_blank_number: Return the number of blank elements in a dataset.
gal_dimension_dist_radial: Radial distance between two coordinates.
gal_fits_key_date_to_struct_tm: FITS date format to C broken-down time.
gal_fits_key_date_to_seconds: FITS date format to Unix epoch time.
gal_qsort_index_single_TYPE_i: Set of functions to sort indexs ascending.
gal_qsort_index_single_TYPE_d: Set of functions to sort indexs descending.
gal_statistics_outlier_cumulative: Uses flatness of the cumulative
distribution to find outliers.
gal_table_list_of_indexs: returns the list of indexs matching columns.
gal_type_is_int: to see if we have an integer (any width, any sign).
** Removed features
** Changed features
Arithmetic:
- 'num' operator is renamed to 'number'.
- 'numvalue' operator is renamed to 'numbervalue'.
- '--dontdelete' will append the output to any existing file. Note that
this change is only in Arithmetic, other programs will still just
complain and abort.
ConvertType:
--forcemin & --forcemax: until now, '--flminbyte' and '--flmaxbyte' were
used to force the range of conversion to color channels (even if the
range is beyond the limits of the dataset). With the introduction of
color maps in Gnuastro 0.8, it is also necessary to force a range on
non-byte datasets. It is thus necessary to use a more generic name.
MakeCatalog:
--std: Until now, this option would measure the mean standard deviation
under the label. But this is not a statistically meaningful measure
for the Sky standard deviation and could be incorrectly used. From now
on, this option measures the square root of the mean variance, or root
mean square (the correct definition of the Sky standard deviation).
NoiseChisel:
--ignoreblankintiles: Until now '--ignoreblankinsky', would specify if
blank values should also be written into the tiled Sky and Sky
standard deviation outputs. But NoiseChisel can optionally produce
many more tiled outputs (for example with '--checkqthresh'). So the
option was renamed to '--ignoreblankintiles' to highlight that the
status of blank elements can be set in all tiled outputs.
Statistics:
--ignoreblankintiles: similar to same option in NoiseChisel.
Table:
--colinfoinstdout: now corresponds to the '-O' short option. Until this
version, the '-s' short option was used for it. But with the new
'--sort' option, '-s' may cause confusion.
Library
gal_arithmetic: new argument: number of threads to use (when relevant).
gal_eps_write: new argument: optional bit-optimization with 'dontoptimize'.
gal_pdf_write: new argument: optional bit-optimization with 'dontoptimize'.
** Bugs fixed
bug #55313: Fits program writing --write values in reverse order
bug #55333: Fits program crash when --write or --update have no value.
bug #55439: Arithmetic segmentation fault when reusing dataset name.
bug #55478: Memory mapping crashes when .gnuastro is not writable.
bug #55491: NoiseChisel crash when no tiles good for quantile thresholding.
bug #55544: Arithmetic's output WCS with where operator is not as expected.
bug #55740: Diamond shapes in nearst-ngb interpolation affecting NoiseChisel.
bug #55763: Crop not keeping Blank pixels on unsigned types.
bug #55844: WCS library (and thus all programs) can't deal with CROTAi values.
bug #55845: Crash when necessary column not found from standard input.
bug #55988: MakeProfiles segfault when input catalog has no rows.
bug #56001: Bad tile sizes with only one tile and small remainder.
bug #56048: Crash when WCS cannot be written to FITS file.
* Noteworthy changes in release 0.8 (library 6.0.0) (2018-12-28) [stable]
** New features
All programs:
- Input files and parameters written as keywords in the first extension
of output when it is FITS. This is only relevant for some programs,
(for example not the Fits or Table programs).
- Standard input (for example from pipes) is now available to feed input
to all programs that accept plain text input (ConvertType, Convolve,
Match, MakeProfiles, Statistics, Table).
- Updated acknowledgment statement (output of '--cite' option).
Arithmetic:
--onedasimage: write output as an image if it has one dimension, not table.
ConvertType:
--colormap: color map to display a single-channel dataset (for example
FITS image) with a range of colors in formats that support color
(like JPEG, or PDF). Until now, the only available mapping was
grayscale. Now "Hue, Saturation, Value" (HSV) and "SLS" (from SAO
DS9) colormaps are also supported.
--rgbtohsv: Convert the RGB input channels to HSV (when the output is
in FITS format).
Convolve:
- Convolves 1D arrays (table columns, for example spectra)
also. Therefore two new options have been added to it: '--column'
('-c', similar to other programs that read table columns), and
'--kernelcolumn' (to specify the column of the kernel in its own
file/extension).
Fits:
--numhdus: prints the number of HDUs in the given FITS file.
NoiseChisel:
- New outlier identification algorithm for quantile thresholds. This is
very useful when there are extended and bright sources in the
dataset: the tiles containing very faint signal that pass the general
pixel-value distribution test due to the flatness of the extended
profiles, can be identified and removed as outliers in comparison
with the other passed tiles. The outlier finding algorithm
('gal_statistics_outlier_positive': a new library function) uses the
distribution of distances between the sorted elements and is
configured with these options.
--outliersclip: Sigma-clipping parameters for the process.
--outliersigma: Multiple of sigma to define an outlier.
--blankasforeground: Treat blank elements as foreground (regions above
the threshold) in the binary processing steps: initial erosion
and opening as well as the filling holes and opening of
pseudo-detections. From this version, by default, blank elements
in the dataset are considered to be background, so if a
foreground pixel is touching it, it will be eroded. This option
is irrelevant if the datasets contains no blank elements, but can
help remove false positives that are touching blank elements.
--holengb: allows defining the connectivity of the holes that are
filled when defining pseudo-detections. Until now, this was
hard-wired into the code (=8) and not modifiable at run-time.
--skyfracnoblank: Ignore blank pixels when estimating the fraction of
undetected pixels for Sky estimation. NoiseChisel only measures
the Sky over the tiles that have a sufficiently large fraction of
undetected pixels. This is done to decrease the bias caused by
faint un-detected wings of bright galaxies or stars, see
description of '--minskyfrac' for more. Until now the reference
for this fraction was the whole tile size (irrespective of how
many blank elements it contains). With this option, it is now
possible to ask for ignoring blank pixels when calculating the
fraction. This is useful when blank/masked pixels are distributed
across the image.
Statistics:
- If an input table has only one column, Statistics won't complain and
abort when no '--column' ('-c') is given: there is only one column to
use anyway, so it will be used. In the absence of which column to
use, it will still complain and abort if the input has more than one
column.
- Sky estimation: new outlier estimation algorithm similar to
NoiseChisel.
- Input can be given using the standard input (for example a pipe).
Library:
- gal_blank_flag_apply: Set all flagged/masked elements to blank.
- gal_fits_key_list_reverse: Reverse the given list of FITS keywords.
- gal_fits_key_write_title_in_ptr: Write a two-line title FITS keyword.
- gal_fits_key_write_in_ptr: New name of gal_fits_key_write.
- gal_fits_key_write_version_in_ptr: old gal_fits_key_write_version.
- gal_fits_key_write_config: write key list and version as config header.
- gal_statistics_outlier_positive: Find the first positive outlier.
- gal_txt_stdin_read: Read lines in standard input into a list of strings.
** Removed features
NoiseChisel:
--mirrordist: mean quantile is now used (not mode), see changes below.
--qthreshtilequant: removed due to new outlier rejection algorithm.
Statistics:
--mirrordist: mean quantile is now used (not mode), see changes below.
** Changed features
Arithmetic:
- If the output has one dimension, it will be written as a table, not a
FITS image/array. This can be changed with the new '--onedasimage'
option.
Convolve:
- The short option for '--numchannels' is now '-n'. Until now, it was
'-c', but that would conflict with the short option used for
'--column' in all the other programs that also read from a table.
MakeProfiles:
--mergedsize: new name for the old '--naxis' option. Since the option
names and values are now written into the FITS header of the
output, this option's name would get confused with the mandatory
FITS keyword 'NAXIS'.
NoiseChisel:
- Until now, the mode's quantile was used to identify tiles with no
significant signal. But from this version, the mean's quantile in
each tile is used instead. The reason is that the mean is more
sensitive to outliers (signal). Therefore the old '--modmedqdiff' is
now called '--meanmedqdiff' .
Statistics:
--meanmedqdiff: new name for '--modmedqdiff'. Similar to NoiseChisel.
Library:
- gal_array_read: list of strings (from standard input) acceptable.
- gal_array_read_to_type: list of strings (from stin) acceptable.
- gal_array_read_one_ch: list of strings (from stdin) acceptable.
- gal_array_read_one_ch_to_type: list of strings (from stdin) acceptable.
- gal_data_copy_to_allocated: Also copies string metadata (e.g., name).
- gal_fits_key_write: filename and hdu instead of FITS pointer.
- gal_fits_key_write_version: filename and hdu instead of FITS pointer.
- gal_fits_key_write_filename: write at the top or end of the input list.
- gal_statistics_outlier_positive: Window-size is now adjustable (new argument).
- gal_table_info: list of strings (from stdin) acceptable.
- gal_table_read: list of strings (from stdin) acceptable.
- gal_txt_table_info: list of strings (from stdin) acceptable.
- gal_txt_image_info: list of strings (from stdin) acceptable.
- gal_txt_table_read: list of strings (from stdin) acceptable.
- gal_txt_image_read: list of strings (from stdin) acceptable.
** Bugs fixed
bug #54493: Warp crashes when type isn't set.
bug #54526: Invalid r, q and truncation of point profiles in MakeProfiles.
bug #54579: NoiseChisel pseudo-detection failure when dataset is negative.
bug #54782: Segment's check image not removing sky clumps some tiles.
bug #54810: Arithmetic crash when previously named operand renamed.
bug #55025: MakeCatalog's '--prepforconv' option being ignored.
bug #55079: Blank EPS or PDF page when width options not given.
bug #55157: No sanity check on values given to Crop's --section.
bug #55295: Crash when more than one collapse operator called.
bug #55298: Arithmetic reading 255 on command-line as blank.
* Noteworthy changes in release 0.7 (library 5.0.0) (2018-08-08) [stable]
** New features
Installation:
--enable-debug: debugging flags, no optimization, no shared libraries.
--enable-check-with-valgrind: Run 'make check' tests within Valgrind.
Arithmetic:
- 'set-A': Set a name ('A' in this case) for the popped dataset. This
allows only reading the dataset it into memory once and
possibly using it many times.
- 'fill-holes': Flip background (0) pixels surrounded by foreground (1).
- 'collapse-sum': collapse/remove a dimension by summing over it.
- 'collapse-min': collapse/remove a dimension by using minimum value.
- 'collapse-max': collapse/remove a dimension by using maximum value.
- 'collapse-mean': collapse/remove a dimension by averaging over it.
- 'collapse-number': Number of elements included in the collapse.
CosmicCalculator:
- Default cosmology set to Plank 2018 results (Paper VI).
MakeCatalog:
--minx: minimum position along first FITS axis.
--maxx: maximum position along first FITS axis.
--miny: minimum position along second FITS axis.
--maxy: maximum position along second FITS axis.
Table:
--colinfoinstdout: column information when writing to standard output.
Library:
- gal_array_name_recognized_multiext: If format contains multiple datasets.
- gal_dimension_collapse_sum: collapse/remove a dimension by summing.
- gal_dimension_collapse_mean: collapse/remove a dimension by averaging.
- gal_dimension_collapse_number: collapse/remove a dimension by number.
- gal_dimension_collapse_minmax: collapse/remove a dimension by extremum.
- gal_wcs_remove_dimension: Remove a dimension in the given WCS structure.
** Removed features
** Changed features
Crop:
--checkcenter: the units of value depend on mode (image or WCS).
MakeCatalog:
- '--checkuplim': new name for '--checkupperlimit'.
- '--brightnessnoriver': new name for '--noriverbrightness'.
Library:
- gal_txt_write: new 'colinfoinstdout' argument.
- gal_table_write: new 'colinfoinstdout' argument.
** Bugs fixed
bug #54057: Building failure due to not finding gsl_interp_steffen.
bug #54063: Match tests in make check fail randomly.
bug #54186: MakeCatalog's --checkupperlimit not keeping output's name.
bug #54188: MakeCatalog's Upperlimit not being sigma-clipped properly.
bug #54284: Crop segfault when catalog contains no data.
bug #54285: make check fails if g++ not present.
bug #54286: BuildProgram's configuration file, not built by default.
bug #54297: No Match output when --notmatched called and no match.
bug #54298: Table not writing array when there are no rows.
bug #54312: Crash when CFITSIO doesn't have fits_is_reentrant function.
bug #54346: Non '-I' or non '-L' strings in CPPFLAGS or LDFLAGS cause crash.
bug #54358: Arithmetic's where, not ignoring blank values in condition.
bug #54406: Insufficient sanity checks in mode symmetricity calculation.
bug #54430: BuildProgram fails with older Libtool versions on dash.
* Noteworthy changes in release 0.6 (library 4.0.0) (2018-06-04) [stable]
** New features
Building:
- New optional dependency: The TIFF library (libtiff).
All programs:
- Input image dataset(s) can be in any of the formats recognized by
Gnuastro (e.g., FITS, TIFF, JPEG), provided that their libraries
(optional dependencies) were present at installation time.
New program:
- Segment: new program in charge of segmentation over detections. This
operation was previously done by NoiseChisel. NoiseChisel is now ONLY
in charge of detection.
Arithmetic:
- erode: Erode the foreground of a binary dataset.
- dilate: Dilate the foreground of a binary dataset.
- filter-sigclip-mean: sigma-clipped, mean filter operator.
- filter-sigclip-median: sigma-clipped, median filter operator.
- connected-components: label the connected elements of the input.
- invert: subtract the maximum of unsigned types (absorption to emission).
- interpolate-medianngb: Interpolate (only blanks) with nearest neighbors.
ConvertType:
- TIFF images can also be used as input.
MakeCatalog:
MakeCatalog will only read the datasets necessary for the requested
columns. Until now, it would read all the possible datasets and all the
intermediate measurements. This is thus major improvement in memory and
CPU usage. As a result, the input argument is no longer assumed to be
the values file, but the object labels file. Please see the
"MakeCatalog inputs and basic settings" section of the book for
more. Here is the summary of the new options:
--insky: Sky value as a single value or file (dataset).
--instd: Sky standard deviation as a single value or file (dataset).
--valuesfile: filename containing the values dataset.
--valueshdu: HDU/extension containing the values dataset.
--clumpscat: Make a clumps catalog also.
--noclumpsort: Don't sort the clumps catalog by host object ID.
--subtractsky: Subtract the given Sky from the values dataset.
--variance: input standard deviation image is actually variance.
--checkupperlimit: make a table for random positions and measurements.
--geoarea: the number of labeled pixels (irrespective of value).
--brightnesserr: error in estimating the brightness.
--mean: calculate the mean pixel value within an object or clump.
--median: calculate the median pixel value within an object or clump.
--upperlimitsigma: position in random distribution (in units of sigma).
--upperlimitquantile: position in random distribution (quantile).
--upperlimitonesigma: 1sigma value of the random distribution.
--upperlimitskew: (mean-median)/sigma or skewness of random distribution.
NoiseChisel:
- New tutorial on detecting large and extended targets.
--rawoutput: only output the detection labels and Sky and its STD.
--ignoreblankinsky: don't set the pixels that are blank in the input to
blank in the Sky and Sky standard deviation outputs (when
'--oneelempertile' is not called).
--label: label the connected detections. Until now this was the default
behavior. However, from this release, NoiseChisel is only in charge
of detection. Segmentation is done by a new program (Segment). Since
detection is ultimately just a binary operator, the default output
now has an 8-bit unsigned integer type with values of 0 or 1. With
this option, you can ask for it to label/count the connected
detections instead of the default binary output.
Statistics:
--manualbinrange: histogram or CFP range can be outside of distribution.
--ignoreblankinsky: similar to same option in NoiseChisel.
Libraries:
gal_array_read: read array from any of known formats (FITS, TIFF, JPEG,...).
gal_array_read_to_type: similar to 'gal_array_read', but to given type.
gal_array_read_one_ch: Read a dataset, abort if it has multiple channels.
gal_array_read_one_ch_to_type: Make sure input is in one channel and type.
gal_binary_label_holes: label the holes within the foreground.
gal_blank_is: check to see if argument is blank in its type or not.
gal_eps_name_is_eps: Returns 1 if given filename is EPS.
gal_eps_suffix_is_eps: Returns 1 if given suffix is EPS.
gal_eps_to_pt: Converts dataset size to PostScript points.
gal_eps_write: Writes a dataset into an EPS file.
gal_interpolate_1d_blank: Fill blank elements through interpolation.
gal_interpolate_1d_make_gsl_spline: Allocate and initalize 'gsl_spline'.
gal_jpeg_name_is_jpeg: Returns 1 if given filename is JPEG.
gal_jpeg_suffix_is_jpeg: Returns 1 if given suffix is JPEG.
gal_jpeg_read: Reads input JPEG image into 'gal_data_t'.
gal_jpeg_write: Writes a 'gal_data_t' into a JPEG file.
gal_label_grow_indexs: grow known indexs into desired areas.
gal_label_watershed: apply watershed algorithm on desired region.
gal_label_clump_significance: measure significance of all clumps in region.
gal_pdf_name_is_pdf: Returns 1 if given filename is PDF.
gal_pdf_suffix_is_pdf: Returns 1 if given suffix is PDF.
gal_pdf_write: Writes a dataset into an PDF file.
gal_pointer_allocate_mmap: Allocate space in a file, not in RAM.
gal_qsort_index_single_d: Sort indexs of single array in decreasing order.
gal_qsort_index_single_i: Sort indexs of single array in decreasing order.
gal_qsort_index_multi_d: Sort indexs in multiple arrays (different threads).
gal_qsort_index_multi_i: Sort indexs in multiple arrays (different threads).
gal_tiff_name_is_tiff: check if name contains a TIFF suffix.
gal_tiff_suffix_is_tiff: check if suffix is a TIFF suffix.
gal_tiff_dir_string_read: convert a string to a TIFF directory number.
gal_tiff_read: Read the contents of a TIFF "directory" to 'gal_data_t'.
** Removed features
NoiseChisel:
- Segmentation (and thus the options below) moved to the new Segment
program: --onlydetection, --segsnminarea, --checkclumpsn, --segquant,
--keepmaxnearriver, --gthresh, --minriverlength, --objbordersn,
--grownclumps, --checksegmentation.
--skysubtracted: no longer necessary, included in noise measuremnts.
MakeCatalog:
--skysubtracted: no longer necessary, included in noise measuremnts.
Library:
- The macros 'GAL_STATISTICS_SORTED_NOT',
'GAL_STATISTICS_SORTED_INVALID', 'GAL_STATISTICS_SORTED_INCREASING',
'GAL_STATISTICS_SORTED_DECREASING': these macros are removed because
we already have the 'GAL_DATA_FLAG_SORT*'' bit-flags in 'gal_data_t'.
** Changed features
Arithmetic:
- filter-mean: a blank value in the input can be non-blank in the
output when non-blank elements present in filter.
- filter-median: similar to filter-mean.
Fits:
--history: can be called/written multiple times in one run.
--comment: can be called/written multiple times in one run.
MakeCatalog:
- The 'WCLUMPS' keyword in the objects labeled image is no longer used
to see if a clumps catalog should also be made. To build a clumps
catalog, you can now use the '--clumpscat' option.
- Estimation of noise-level is now done per-pixel over the whole
label. Until now the average noise level was used.
--objectsfile has been removed. The main input argument is now assumed
to be the objects file.
NoiseChisel:
From this release, NoiseChisel is only in charge of detection and won't
do segmentation any more. The new Segment program is in charge of
segmentation. Many of the changes below are due to this new limited
scope of NoiseChisel.
--kernel: value 'none' will disable convolution.
- Renamed options:
--convolvedhdu ==> --chdu
--wkhdu ==> --whdu
--detsnminarea ==> --snminarea
--checkdetsn ==> --checksn
--detquant ==> --snquant
- By default the output detection map is a binary image (values of 0 or 1).
- With no output name, the output has a '_detected.fits' suffix.
Segment:
- [Previously in NoiseChisel]: For finding true clumps, the difference
in the peak of the clump and the highest valued river pixel, divided
by the noise standard deviation are used. Until now, the total
signal-to-noise ratio was used as a criteria. In initial tests, this
algorithm was much more promising in detecting clumps over strong
gradients and also on flatter gradients.
Table:
--column: multiple columns (comma separated) can be used in one
instance of this option (multiple instances of this option are
still acceptable also).
Libraries:
gal_binary_holes_fill: new name for 'gal_binary_fill_holes'.
gal_dimension_is_different: new name for 'gal_data_dsize_is_different'.
gal_fits_img_read: now only reads the data not the WCS, therefore it no
longer needs the last two arguments. A subsequent call to
'gal_wcs_read' can be used to read the WCS information in the file.
gal_pointer_increment: new name for 'gal_data_ptr_increment'.
gal_pointer_num_between: new name for 'gal_data_ptr_dist'.
gal_pointer_allocate: replaces 'gal_data_malloc_array' and
'gal_data_calloc_array', through an argument you can ask for the
allocated memory to be cleared or not.
gal_qsort_TYPE_i: new name for gal_qsort_TYPE_increasing.
gal_qsort_TYPE_d: new name for gal_qsort_TYPE_decreasing.
gal_statistics_is_sorted: can now also update the bit-flags regarding
the sorted nature of the input (to optimize future calls to the
function).
gal_statistics_quantile_function: returns 'inf' or '-inf' if the given
value is smaller than the minimum or larger than the maximum of the
input dataset's range. Until now, it would return blank in such
cases.
gal_statistics_number: the output dataset now has a 'size_t' type. Until
now it was 'uint64_t'.
** Bugs fixed
bug #50957: --version output not possible on Mac OS X
bug #52979: Many unused result warnings for asprintf in some compilers.
bug #53122: Configure time CPPFLAGS and LDFLAGS don't pass to BuildProgram.
bug #53142: Crash when printing values with the '--onlyversion' option.
bug #53147: NULL value of onlyversion option causing a crash.
bug #53226: Match output directory ignored when making multiple files.
bug #53230: Statistics program bad results on integer columns with limits.
bug #53268: NoiseChisel crash when no growth is possible.
bug #53295: MakeCatalog parses area larger than clump.
bug #53304: NoiseChisel crash when there is no detection.
bug #53312: Fits crash on keyword editing (except --delete).
bug #53407: Instrumental noise in MakeNoise should be squared.
bug #53424: Sigma-clipping seg-faults when there are no more elements.
bug #53580: Warp crash when no WCS present.
bug #53825: NoiseChisel not accounting for fully zero-valued tiles.
* Noteworthy changes in release 0.5 (library 3.0.0) (2017-12-22) [stable]
** New features
Manual/Book: An extended tutorial is added showing some general
applications of almost all the programs. This may be a good place to get
a feeling of how Gnuastro is intended to be used and some of the
programs.
New Program and library: Match is a new program that will match two given
inputs (currently catalogs). Its output is the re-arranged inputs with
the same number of rows/records such that all the rows match. The main
work is also done with the new low-level 'gal_match_catalog' library
function which can also be used in more generic contexts.
All programs: a value of '0' to the '--numthreads' option will use the
number of threads available to the system at run time.
Arithmetic: The new operators 'filter-median' and 'filter-mean' can be
used to filter (smooth) the input. The size of the filter can be set as
the other operands to these operators.
BuildProgram: The new '--la' option allows the identification of a
different Libtool '.la' file for Libtool linking information.
BuildProgram: The new '--deletecompiled' option will delete the compiled
program after running it.
CosmicCalculator: all the various cosmological calculations can now be
requested individually in one line with a specific option added for each
calculation (for example '--age' or '--luminositydist' for the age of the
universe at a given redshift or the luminosity distance). Therefore the
old '--onlyvolume' and '--onlyabsmagconv' options are now removed. To
effectively use these new features, please review the "Invoking
CosmicCalculator" section of the book.
Fits: when an extension/HDU is identified on the command-line with the
'--hdu' option and no operation is requested, the full list of header
keywords in that HDU will be printed (as if only '--printallkeys' was
called).
MakeCatalog: physical nature agnostic WCS column names. Previously the
first WCS axis was always assumed to be RA and the second DEC. So for
example even if you had a spectrum (with X and wavelength as the two WCS
dimensions), you would have to ask for '--ra' and '--dec'. The new '--w1'
and '--w2' options are now generic and don't assume any particular type
only their order in the FITS header. MakeCatalog now also uses the CTYPE
and CUNIT keywords to set the names and units of its output columns. The
'--ra' and '--dec' options are now just internal aliases for '--w1' or
'--w2' which will be determined based on the input's CTYPE keyword. Also
the new '--geow1', '--geow2', '--clumpsw1', '--clumpsw2',
'--clumpsgeow1', '--clumpsgeow2' options replace the old options
'--geora', '--geodec', '--clumpsra', '--clumpsdec', '--clumpsgeora',
'--clumpsgeodec'. No alias is currently defined for the latter group.
MakeCatalog: the new '--uprange' option allows you to specify a range for
the random values around each object. This is useful when the noise
properties of the dataset vary gradually and sampling from the whole
dataset might produce biased results.
NoiseChisel: with the new '--convolved' and '--convolvedhdu' options,
NoiseChisel will not convolve the input any more and use the given
dataset instead. In many cases, as the inputs get larger, convolution is
the most time consuming step of NoiseChisel. With this option, you can
greatly speed up your tests (to find the best parameters by varying them,
for a given analysis). See the book for more information and examples.
NoiseChisel: with the new '--widekernel' option it is now possible to use
a wider kernel to identify which tiles contain signal. The rest of the
steps (identifying the quantile threshold on the selected tiles and etc)
are done on the dataset convolved with '--kernel' as they were
before. Since it is time consuming, this is an optional feature.
NoiseChisel: with the new '--qthreshtilequant' option, it is now possible
to discard high-valued (outlier) tiles before estimating qthresh over the
whole image. This can be useful in detecting very large diffuse/flat
regions that would otherwise be detected as background (and effectively
removed).
NoiseChisel: the finally selected true detections are now grown based on
signal contiguity, not by blind dilation. The growth process is the same
as the growing of clumps to define objects. Only for true detections, the
growth occurs in the noise. You can configure this growth with the
'--detgrowquant' and '--detgrowmaxholesize'. With this new feature it is
now possible to detect signal out to much lower surface brightness limits
and the detections don't look boxy any more.
Cosmology library: A new set of cosmology functions are now included in
the library (declared in 'gnuastro/cosmology.h'). These functions are
also used in the CosmicCalculator program.
'gal_table_read' can now return the number of columns matched with each
input column (for example with regular expressions), a new argument has
been added to allow this feature.
'gal_fits_key_img_blank': returns the value that must be used in the
BLANK keyword for the given type as defined by the FITS standard.
'gal_txt_write' and 'gal_fits_tab_write' now accept an extension name as
argument to allow a name for the FITS extension they write.
'gal_box_bound_ellipse_extent' will return the maximum extent of an
ellipse along each axis from the ellipse center in floating point.
** Removed features
Installation: The '--enable-bin-op-*' configuration options that were
introduced in Gnuastro 0.3 have been removed. By managing the arithmetic
functions in a better manner (a separate source file for each operator),
compilation for all types (when done in parallel) takes about the same
time as it took with the default (only four) types until now.
MakeCatalog: '--zeropoint' option doesn't have a short option name any
more. Previously it was '-z' which was confusing because '-x' and '-y'
were used to refer to image coordinate positions.
NoiseChisel: The '--dilate' and '--dilatengb' options have been
removed. Growing of true detections is no longer done through dilation
but through the '--detgrowquant' and '--detgrowmaxholesize' options (see
above).
** Changed features
CosmicCalculator: The redshift is no longer mandatory. When no redshift
is given, it will only print the input parameters (cosmology) and abort.
MakeCatalog: when the output is a FITS file, the two object and clumps
catalogs will be stored as multiple extensions of a single file. Until
now, two separate FITS files would be created. Plain text outputs are the
same as before (two files will be created).
'gal_binary_fill_holes' now accepts a 'connectivity' and 'maxsize'
argument to specify the connectivity of the holes and the maximum size of
acceptable holes to fill.
'gal_fits_img_read' and 'gal_fits_img_read_to_type' now also read the WCS
structure of the extension/HDU in a FITS file and have two extra
arguments: 'hstartwcs' and 'hendwcs'. With these options it is possible
to limit the range of header keywords to read the WCS, similar to how
they are used in 'gal_wcs_read'.
'gal_txt_write', 'gal_table_write_log', 'gal_fits_tab_write' and
'gal_txt_write' don't have the 'dontdelete' argument any more. The action
they take if the file already exists depends on the file: for FITS, a new
extension will be added and for text, they will abort with an error.
'gal_tile_block_write_const_value' and 'gal_tile_full_values_write' now
accept a new 'withblank' option to set all pixels that are blank in the
tile's block to be blank in the check image.
'gal_wcs_pixel_area_arcsec2' will return NaN (instead of aborting) when
input is unreasonable (not two dimensions or not in units of degrees).
'gal_wcs_world_to_img' and 'gal_wcs_img_to_world': Until now, these two
WCS conversion functions would explicitly assume RA and Dec and work
based on input arrays (so for example it was also necessary to give the
number of elements and etc). They now accept 'gal_data_t' as input for
the input coordinates, thus their API has been greatly simplified and
their functionality increased.
** Bugs fixed
ConvertType crash when changing values (bug #52010).
Arithmetic not accounting for integer blank pixels in binary operators
(bug #52014).
NoiseChisel segfault when memory mapping to a file (bug #52043).
CFITSIO 3.42 and libcurl crash at Gnuastro configure time (bug #52152).
MakeCatalog crash in upper-limit with full size label (bug #52281).
NoiseChisel leaving unlabeled regions after clump growth (bug #52327).
Arithmetic crash with no input tokens (bug #52422).
Libtool checks only in non-current directory (bug #52427).
* Noteworthy changes in release 0.4 (library 2.0.0) (2017-09-13) [stable]
** New features
All programs: '.fit' is now a recognized FITS file suffix.
All programs: ASCII text files (tables) created with CRLF line
terminators (for example text files created in MS Windows) are now also
readable as input when necessary.
Arithmetic: now has a new '--globalhdu' ('-g') option which can be used
once for all the input images.
MakeNoise: with the new '--sigma' ('-s') option, it is now possible to
directly request the noise sigma or standard deviation. When this option
is called, the '--background', '--zeropoint' and other option values will
be ignored.
MakeProfiles: the new '--kernel' option can make a kernel image without
the need to define a catalog. With this option, a catalog (or
accompanying background image) must not be given.
MakeProfiles: the new '--pc', '--cunit' and '--ctype' options can be used
to specify the PC matrix, CUNIT and CTYPE world coordinate system
keywords of the output FITS file.
MakeProfiles: the new 'distance' profile will save the radial distance of
each pixel. This may be used to define your own profiles that are not
currently supported in MakeProfiles.
MakeProfiles: with the new '--mcolisbrightness' ("mcol-is-brightness")
option, the '--mcol' values of the catalog will be interpretted as total
brightness (sum of pixel values), not magnitude.
NoiseChisel: with the new '--dilatengb' option, it is now possible to
identify the connectivity of the final dilation.
Library: Functions that read data from an ASCII text file
('gal_txt_table_info', 'gal_txt_table_read', 'gal_txt_image_read') now
also operate on files with CRLF line terminators.
** Removed features
** Changed features
Crop: The new '--center' option is now used to define the center of a
single crop. Hence the old '--ra', '--dec', '--xc', '--yc' have been
removed. This new option can take multiple values (one value for each
dimension). Fractions are also acceptable.
Crop: The new '--width' option is now used to define the width of a
single crop. Hence the old '--iwidth', '--wwidth' were removed. The units
to interpret the value to the option are specified by the '--mode'
option. With the new '--width' option it is also possible to define a
non-square crop (different widths along each dimension). In WCS mode, its
units are no longer arcseconds but are the same units of the WCS (degrees
for angles). '--width' can also accept fractions. So to set a width of 5
arcseconds, you can give it a value of '5/3600' for the angular
dimensions.
Crop: The new '--coordcol' option is now used to determine the catalog
columns that define coordinates. Hence the old '--racol', '--deccol',
'--xcol', and '--ycol' have been removed. This new option can be called
multiple times and the order of its calling will be used for the column
containing the center in the respective dimension (in FITS format).
MakeNoise: the old '--stdadd' ('-s') option has been renamed to
'--instrumental' ('-i') to be more clear.
MakeProfiles: The new '--naxis' and '--shift' options can take multiple
values for each dimension (separated by a comma). This replaces the old
'--naxis1', '--naxis2' and '--xshift' and '--yshift' options.
MakeProfiles: The new '--ccol' option can take the center coordinate
columns of the catalog (in multiple calls) and the new '--mode' option is
used to identify what standard to interpret them in (image or
WCS). Together, these replace the old '--xcol', '--ycol', '--racol' and
'--deccol'.
MakeProfiles: The new '--crpix', '--crval' and '--cdelt' options now
accept multiple values separated by a comma. So they replace the old
'--crpix1', '--crpix2', '--crval1', '--crval2' and '--resolution'
options.
'gal_data_free_contents': when the input 'gal_data_t' is a tile, its
'array' element will not be freed. This enables safe usage of this
function (and thus 'gal_data_free') on tiles without worrying about the
memory block associated with the tile.
'gal_box_bound_ellipse' is the new name for the old
'gal_box_ellipse_in_box' (to be more clear and avoid repetition of the
term 'box'). The input position angle is now also in degrees, not
radians.
'gal_box_overlap' now works on data of any dimensionality and thus also
needs the number of dimensions (elements in each input array).
'gal_box_border_from_center' now accepts an array of coordinates as one
argument and the number of dimensions as another. This allows it to work
on any dimensionality.
'gal_fits_img_info' now also returns the name and units of the dataset
(if they aren't NULL). So it takes two extra arguments.
'gal_wcs_pixel_scale' now replaces the old 'gal_wcs_pixel_scale_deg',
since it doesn't only apply to degrees. The pixel scale units are defined
by the units of the WCS.
'GAL_TILE_PARSE_OPERATE' (only when 'OTHER' is given) can now parse and
operate on different datasets independent of the size of allocated block
of memory (the tile sizes of 'IN' and 'OTHER' have to be identical, but
not their allocated blocks of memory). Until now, it was necessary for
the two blocks to have the same size and this is no longer the case.
** Bugs fixed
MakeProfiles long options on 32bit big endian systems (bug #51341).
Pure rotation around pixel coordinate (0,0) (bug #51353).
NoiseChisel segfault when no usable region for sky clumps (bug #51372).
Pixel scale measurement when dimension scale isn't equal or doesn't
decrease (bug #51385).
Improper types for function code in MakeProfiles (bug #51467).
Crashes on 32-bit and big-endian systems (bug #51476).
Warp's align matrix when second dimension must be reversed (bug #51536).
Reading BZERO for unsigned 64-bit integers (bug #51555).
Arithmetic with one file and no operators (bug #51559).
NoiseChisel segfault when detection contains no clumps (bug #51906).
Correct size checking when allocating gal_data_t (bug #52544).
* Noteworthy changes in release 0.3 (library 1.0.0) (2017-06-01) [stable]
This is a full re-write of Gnuastro. Most importantly, Gnuastro now has a
new generic data container ('gal_data_t'). This new container can now
deal natively with all standard numeric data types, work in RAM or
HDD/SSD, keep data in any dimensions and has enabled many other very
useful features. Some of the most prominent of the new features are
discussed below. It is strongly recommended to review the respective
section of the Gnuastro manual/book for a better feeling of all the new
features.
As discussed below, some program names have changed, if you have a
previous version of Gnuastro installed from source, it is recommended to
uninstall it first (with 'make uninstall' using the corresponding
tarball), then install this new version. Building Gnuastro can be slow,
so please build in parallel with Make's '-j8' option (to build on 8
threads).
** New programs or library features
Library functions that deal with datasets now use this generic data
container for inputs and outputs, significantly simplifying their
API. Nearly all library functions have been re-written with much more
clear names, argument lists and individual purpose. Some example library
functions are shown below, also see the "Library demos" section of the
book for some complete working example:
-- 'gal_table_read' and 'gal_table_write' will read and write data to
plain text, FITS ASCII and FITS Binary formats.
-- 'gal_fits_img_read' and 'gal_fits_img_write' can read a FITS image
to memory or write a FITS image from memory.
Gnuastro now defines a simple comment line format to keep basic
information in a plain text table, see the "Gnuastro text table format"
section of the book. In short for every column, a comment line like below
can be used to give a name, units, comments, or a type to a column. This
allows a FITS binary table for example to be written to plain text and
converted back to binary without loosing any information (except for very
small floating point errors if not enough decimals are printed).
# Column N: NAME [UNIT, TYPE, BLANK] COMMENT
The new Fits program replaces the old Header program. But besides
reading/checking FITS header keywords, it can now also work on FITS
extensions/HDUs. For example with no options, it will list all the HDUs
in a FITS file along with basic information. It can copy a whole HDU to
another file, or delete a HDU from a FITS file. To get the previous
behavior of listing all the keywords in a FITS HDU, you can run it with
the '-p' option.
All programs now write data into the second HDU of a FITS file to allow a
clean first HDU. Note that following CFITSIO, HDU counting still starts
from zero, so FITS images and tables written by Gnuastro in a new file
can always be accessed with the '--hdu=1' option (which is now the
default).
If any program is run within a Git version controlled directory, a
'COMMIT' header keyword will be added to the created FITS files, see the
"Output headers" section of the book for a discussion on the usefulness
of this new feature.
BuildProgram: a new program to easily compile, link and run a C program
you have written with Gnuastro's libraries without having to worry about
which libraries (Gnuastro dependencies) your program needs. Debugging
('-g'), optimizations ('-O'), warnings ('-W'), include search path
('-I'), link search path ('-L'), and linked libraries ('-l') compiler
options are also supported. BuildProgram will greatly facilitate the easy
usage of Gnuastro's libraries.
** Removed programs or library features
The following program names have been renamed:
ImageCrop --> Crop
ImageWarp --> Warp
ImageStatistics --> Statistics
Header --> Fits
SubtractSky has been removed. The Statistics program now has tools to
estimate the Sky value and it can be subtracted with Arithmetic.
** New features
All Gnuastro programs that read and write tables can now do so in plain
text table format or in FITS ASCII or FITS Binary tables. Depending on
the filename or with the new '--tableformat' common option to all
programs.
The option management system in all Gnuastro programs has been completely
re-written with many new features, some of the most important ones are
listed below. For developers, you will notice that there is no more usage
of macros and adding new options has become much more easier.
-- All programs will now also look for a 'gnuastro.conf' configuration
file to keep common options for all programs in every directory.
-- The '--lastconfig' option can be used on the command-line or in any
configuration file to stop parsing any further configuration files.
-- The '--config' option can now be used to identify any arbitrary
file to be parsed as a configuration file. Any file that is given
to this option is parsed immediately.
-- The '--printparams' option now also prints the short documentation
of each option (same description in '--help') after its value.
It is now possible to choose columns in tables based on column names as
well as column numbers. It is also possible to search for columns based
on searching in their units or comments. When using column numbers,
counting now starts from 1 (one), not 0 (as before). See the new
"Selecting table columns" section for more on these new features.
Where relevant, all programs now accept a '--type' option that you can
use to specify the numerical datatype of the output.
With the new common option '--minmapsize', you can specify a minimum size
of an array (in bytes) to store data in SSD/HDD and not in RAM. This can
be instrumental when you are dealing with large datasets, or even smaller
ones, but when your RAM is getting full.
Making a log file is now optional and users have to explicitly ask for it
with the '--log' option.
Slower building of Gnuastro: Binary operators (e.g., plus or multiply)
are now done in the native type of the input dataset. Doing so for all
the different combinations of types, greatly slows down the initial
compilation of Gnuastro (after running 'make'). So for every type there
is now a '--enable-bin-op-*' configure time option. When the dataset's
type isn't compiled (only for the binary operators), it will be converted
to a compiled type and then converted back in the end.
Arithmetic: all operations are done in the native data type of the
dataset. Until now, it would convert the data internally to double
precision floating point, do the requested operation and write the data
back in the proper type. So this new implementation is much more
efficient.
Arithmetic: two new classes of operators: type conversion operators to
all standard types and integer-only operators (all C bit-wise operators
and the modulo operator).
ConvertType: can also print the input dataset to the command-line
('stdout'). To use this feature set the output filename to 'stdout'.
Convolve now has the '--minsharpspec' option to specify the minimum
spectrum value to use in deconvolution (matching PSFs).
Crop: when in WCS mode it can still only work on aligned images. However,
very small floating point errors in writing the WCS (for example 10e-13
degrees) are now acceptable. Until now these would cause Crop to complain
and abort.
Crop: the name of the crop can be pre-determined based on the values in a
given in a table column. This allows your object's IDs to be directly
used as the crop's file name for example.
MakeCatalog now also reports the surface brightness in
mag/arcsec^2. Until now, MakeCatalog would only print the dataset's
surface brightness in units of magnitudes/pixel. But that is not nicely
comparable to other datasets. Hence, using a simple calculation (from the
pixel projected size, fully derived in the book), it now also reports the
surface brightness in magnitudes/arcsec^2 also.
MakeProfiles: Profile codes now start from '1' (until now they started
from '0').
MakeProfiles: now accepts the radial function of profiles as
human-readable strings instead of a code for each profile (which was very
cryptic, although codes are also still acceptable). For example in the
profile column you can now write 'sersic' instead of the code '1'.
NoiseChisel: the new '--cleandilated' option will remove dilated objects
that have a low S/N (it is mainly useful on very clean or mock images).
For non-clean noise, it will result in a decrease of completeness. With
this new option, NoiseChisel will also print detection S/N values when
run with the '--checkdetsn' option.
Statistics: now reads table columns as well as images and does basic
operations on them. It can also only work on a certain range of the data
(instead of the whole set). Alternatively, you can define the range on
another reference column, but use values of the main column.
Statistics: all its single-valued measurements can now be done on a
tessellation (tile grid) over the input dataset.
Statistics: can now estimate the Sky value on the input dataset using the
mode's quantile similar to what NoiseChisel does to find its initial
threshold.
Statistics: has several new single valued calculations: '--quantile',
'--quantfunc' (quantile function), '--mode', '--modequant', '--modesym',
and '--modesymvalue'.
Warp: align the image with the celestial coordinates using the '--align'
option.
Warp: standard modular warpings can now be requested without an input
matrix, using the following options: '--shear', '--flip', '--project',
'--rotate', '--scale', '--translate'. Any number of these transformations
(along with the '--align' option) can be called on the command-line and
they will be applied in the same order to create one warping matrix. By
default the WCS will also be corrected.
** Changes in behavior
Mask image options have been removed from all programs. Instead, all
programs can work directly on data with blank values. So when some pixels
must be masked, the Arithmetic program's 'where' operator can be used to
select special pixels and set them to blank. In particular bit-wise
operations are now available in Arithmetic to use bit-mask
images. Managing all these different choices in every program would only
confuse the user (with too many options).
Arithmetic: the 'x' letter is now used to represent the multiplication
operator. Previously it was '*' which needed quotation and was thus very
inconvenient.
Convolve: the old '--frequency' and '--spatial' options have been removed
and are replaced by '--domain' which accepts values of 'frequency' and
'spatial'.
Convolve: the old '--viewfreqsteps' was changed to '--checkfreqsteps' to
fit with the general style of such check images in all Gnuastro's
programs.
Crop: '--section' syntax is now inclusive in both bounds.
Crop: only checks if the center of a crop is filled when the crop was
defined by its center (for example with '--ra' and '--dec'). The verbose
outputs of Crop are also not cryptic 0s or 1s. The are human readable
text.
Crop: doesn't have separate '--imgmode' and '--wcsmode' options any
more. There is now a single '--mode' option which accepts values of 'img'
or 'wcs'.
MakeProfiles: the old '--inputascanvas' is now called '--clearcanvas'.
MakeProfiles: until now, it would abort with an error when the input
columns had blank values. But for masking, it might happen that you set a
blank magnitude. So this check has now been removed when reading the
magnitude column.
NoiseChisel: default value of the '--minskyfrac' option (new name for the
old '--minbfrac') is now 0.7 as opposed to 0.5. This will allow much
better estimation of noise properties (by default). It may be slightly
too high for a crowded field, but the users can change it on the
command-line (or in a configuration file) for such datasets.
NoiseChisel: when it is run with any of the '--check' options, it will
abort after all the check images have been created. This is very useful
for checking your parameters until each step and not be distracted (or
have to wait) for later steps to finish.
Statistics: will not make a histogram and cumulative frequency files, or
calculate sigma-clipped results by default (with no options). It will
just print some basic information.
Table: Previously, if a column was requested, the '-i' option would be
ignored. But it often happens that the users forget a column name after
already typing several of their desired columns. So the opposite behavior
is preferred. Because when more than a couple of columns are needed, you
will probably forget the column identifiers of the last few and having to
retype everything is very frustrating. Something like how '--help' takes
precedence over all other options.
Table: to select column(s) by regular expression searching, the name now
has to be put in '/ /' (similar to AWK). If a value isn't in '/ /', the
programs will only select a column with the exact match.
Warp: when a 2 by 2 matrix is given, the FITS pixel positions (which
define the center of a pixel as an integer) are automatically implemented
internally, see "Invoking Warp" in the manual for more.
Warp: the old '--nofitscorrect' option has been changed to
'--centeroncorner' to be more clear. The new option is now more general
than before and also works on warping with a matrix, not just on modular
warpings.
Warp: the old '--nowcscorrection' option has been given a more clear name
of '--keepwcs'. With this option, Warp will not apply the warp the
input's WCS structure.
Warp: the old '--maxblankfrac' option has been changed to
'--coveredfrac'. Until now, Warp would only look for the fraction of
input blank/NaN pixel area over the output pixel. But this would be
useless on the edges of the image. So the new '--coveredfrac' option
takes the acceptable fraction of output pixel area that must be covered
by input pixels in order to give that output pixel a value. You can use
this to set edge pixels that are not fully covered in the new grid to
blank and have a flat warped image.
** Bugs fixed
Using '%zu' to print 'size_t' variables for clean build on 32-bit
systems.
Crash in Table for some operating systems due to memory is now fixed (bug
#49347).
Table's man-page is now created and installed (bug #49418).
Fixes in the documentation (sr #109170, bug #49419).
Check for malloc returning valid pointer (bug #49459).
Segfault in mesh interpolation corrected (bug #49588).
Corrected bad status usage in calls to wcsp2s and wcss2p (bug #49752).
Stricter checking in Crop's polygon point list (bug #48978).
Correction in alignment and getting pixel scale (bug #50072).
Decomposing PCi_j and CDELTi matrices in output WCS (bug #50073).
Using image naxes[n] when checkcenter is larger (bug #50099).
Memory leak in MakeCatalog corrected (bug #51118).
Fix copy-paste error in MakeCatalog flag arrays (bug #51130).
* Noteworthy changes in release 0.2 (library 0.0.0) (2016-10-03) [stable]
** Bugs fixed
Linker errors on some operating systems have been fixed (bug #48076).
Several memory allocation, checks or redundancies have been fixed: bugs
#48453, #48516, #48603, #48611, #48571, #48650, #48657, #48692, #48770,
#47866, #48899, #49049, #49007.
ImageCrop no longer crashes with very long output file names (bugs #46241
and #45380).
** New programs or headers
Table: a new utility to read and write FITS binary and ASCII tables. It
can also print the column information or select columns using regular
expressions (task #13579).
Shared libraries and headers are now installed. The libraries can be used
in C and C++ programs. This release includes the following headers:
'gnuastro.h', 'array.h', 'box.h', 'fits.h', 'linkedlist.h', 'mesh.h',
'polygon.h', 'qsort.h', 'spatialconvolve.h', 'statistics.h', 'threads.h',
'wcs.h', 'txtarray.h' (task #13765).
Gnuastro now comes with a script in its top source directory
('tmpfs-config-make') to configure and build it in the tmpfs (on the
RAM), for those systems that have it. See the new "Configure and build in
RAM" section in the book for more (task #14100).
** New features
MakeProfiles also accepts WCS positions (task #13566).
Flat profiles in MakeProfiles can be given a profile specific value. The
new '--mforflatpix' option MakeProfile will use the value in the
magnitude column as a fixed value for each pixel. This can be very useful
in defining a mask, or creating segmentation maps or labeled images (task
#14115).
MakeProfiles can now use input image as canvas. Instead of specifying the
WSC and image size parameters manually. With the new '--inputascanvas'
option, MakeProfiles will get this information (along with blank pixels)
from an already existing image (task #14116).
Type of output in MakeProfiles and Arithmetic can be specified with the
'--type' option.
Magnitude error column in MakeCatalog with the '--magnitudeerr' option.
Arithmetic now has new conditional (task #13870) and logical operators
(task #14153) along with an operator for actions only when conditions are
true: 'where'. The new 'isblank' operator will also enable you to select
blank, or masked, pixels (task #14146).
The '--noerodequant' in NoiseChisel enables it to detect small and
sharper profiles by disabling erosion on pixels above a certain quantile
(task #14139).
MakeCatalog can also calculate the upper limit magnitude for each given
object in the image by randomly positioning the object's footprint over
undetected regions of the image (task #14167).
The source tarball is now also distributed with Lzip for a much better
compression ratio and more robust archival file format.
** Changes in behavior
The two MakeProfiles options '--setconsttonan', '--setconsttomin' have
been removed (see '--mforflatpix' above for their alternative).
MakeCatalog makes clump catalog only when asked (when the 'WCLUMPS'
header exists in the objects HDU). This can be very useful in cases like
aperture photometry, when the object labels are not generated by
NoiseChisel and so a clump image doesn't exist (task #14122).
Default cosmological parameters in CosmicCalculator set to Plank 2015
results: A&A (2016), 594, A13 (arXiv 1502.01589).
The '--envseed' option (to read random number generator type and seed
from the environment) to MakeProfiles and MakeNoise can also be given in
the configuration files.
* Noteworthy changes in release 0.1 (2016-05-30) [stable]
** Bugs fixed
MakeCatalog's problem in checking the sizes of all input images is now
fixed.
NoiseChisel's problem with reading the '--kernel' option is now
corrected (bug #46750).
lib/mesh.c's problem in correctly calculating the mesh sizes was
corrected (bug #47611).
'make check' will not look into system utility configuration files. In
the previous release, if Gnuastro was already installed, the
configuration files already present on the system would also be read. Now
only configuration files in the tested package are used (bug #47833).
Ghostscript's version is now checked at configure time after its
existence. ConvertType uses the '-dPDFFitPage' option to Ghostscript
which was introduced in version 9.10, so older versions would pass
configure but at 'make check' time, the PDF test would fail. Now this
test is skipped (bug #47868).
Most tests would fail when 'make check -jN' was run (to do the checks on
N threads). A dependency structure has now been defined to fix this
problem and greatly speed up the testing process (bug #47957).
** New utilities
Arithmetic: For arithmetic operations on the pixels of input images. With
this utility, it is now possible to add multiple images with each other,
or easily calculate a median image. It as a large set of other arithmetic
operations and some functions which can be done on the input image(s). An
unlimited number of input images can be given, the images will only be
loaded when necessary and will be freed as soon as they are no longer
necessary.
CosmicCalculator: For doing cosmological calculations at a given
redshift.
** New features
All the utilities that would produce a log file now have a '--nolog'
option to avoid printing a log file.
The tiled image compression convention (.fits.fz, created with 'fpack')
files can now be used as input in the utilities.
ImageCrop can now also crop a polygon from the input image. The polygon
vertices can be given in the world or image coordinates. The simple
'--polygon' option will keep the insides of the polygon while the
'--outpolygon' will keep the outside of the polygon.
ImageCrop and ImageWarp can now read the WCS information of a FITS header
from a specific region with the '--hstartwcs' and '--hendwcs'. In some
older FITS images, when the WCS distortions were not as standardized as
now, there were cases which would confuse WCSLIB.
NoiseChisel can now save the grown clumps image instead of the original
clumps image in the output with the '--grownclumps' option.
Convolve can now do deconvolution with the '--makekernel' option.
MakeProfiles now has a '--setconsttonan' option which will fill the
constant profiles with a NaN (blank) value, not a number, allowing the
creations of elliptical masked regions for example.
Header can now import a keyword directly from a string with the '--asis'
option.
MakeCatalog can now output the geometric (average position independent of
pixel flux value) positions of the objects too.
MakeCatalog can now produce the object's elliptical parameters (for
example semi-major axis, semi-minor axis, and position angle). This can
also be done both in standard flux weighted and geometric methods too.
MakeCatalog now has a '--threshold' function to only use pixels above a
given threshold in each object or clump. This is useful to avoid diffuse
regions in calculations.
MakeCatalog now has a '--noriverbrightness' option. With this option it
is possible to calculate the clump flux without subtracting the river
pixels on its circumference.
The number of CPU threads is no longer a configuration option, it is now
determined at run-time for each program. Therefore it is now easily
possible to built Gnuastro on one system to use on another (commonly done
in the GNU/Linux package managers). Therefore ./configure no longer has a
'--with-numthreads' option.
Every commit in Gnuastro's history (after implementing this feature) can
now be given a unique version number. Since the version number is printed
in possible outputs, this feature can help reproducibility, even when the
an official/stable release isn't used.
The AUTHORS file is now automatically generated from the version
controlled history. Also all the authors that have contributed to
Gnuastro are included in the second (copyright) page of the PDF book.
All the bootstrapped directories are now moved with a new 'bootstrapped'
directory in the top source directory. This significantly cleans up this
directory, allowing users to more easily find the hand-written Gnuastro
source files they like.
A 'bug-gnuastro' Info page was created so users can easily go to that
page for information on how to submit bug reports. It is accessible on
the command line with the command 'info bug-gnuastro'.
** Changes in behavior
The separate utilities no longer have a separate version number. With the
introduction of unofficial version numbers generated from each commit in
Gnuastro's history, the utility version numbers would cause confusion.
MakeProfiles will now add a suffix to the individual images and put them
in the output directory if specified.
NoiseChisel and MakeCatalog now use the median mesh standard deviation to
define the over-all depth of the image. Previously they used the maximum
value (or the least depth).
NoiseChisel no longer outputs a sky subtracted image. This job can now be
done with the new Arithmetic utility.
NoiseChisel's '--segsnhistnbins' option was renamed to
'--clumpsnhistnbins'.
** Improvements
NoiseChisel's default quantiles were changed to 0.95 from 0.99. Since the
old value was too severe.
NoiseChisel's S/N thresholds are now found from the distribution of
pseudo-detections and clumps from the full image, not within each large
mesh. This was done to increase the accuracy of the S/N threshold. There
were commonly not enough points in large mesh sizes and this would add
scatter. With ImageCrop's polygon capabilities, it is now easily possible
to cut out the region that has uniform noise properties (depth and
correlated noise). Therefore the old '--checkdetectionsn' and
'checkclumpsn' options are no longer present.
When building from the version controlled source, the whole bootstrapping
process is done with one script. In the previous version, all the
separate operations should have been done by hand (as instructed in the
old manual).
All the build steps now report what was done and suggest the next
step. This feature can be disabled with the '--disable-guide-message' at
configure time.
* Copyright notice
Copyright (C) 2015-2025 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document under
the terms of the GNU Free Documentation License, Version 1.3 or any later
version published by the Free Software Foundation; with no Invariant
Sections, with no Front-Cover Texts, and with no Back-Cover Texts.
|