gdal-async

_ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| Changelog Version 7.76.1 (14 Apr 2021) Daniel Stenberg (14 Apr 2021) - RELEASE-NOTES: synced curl 7.76.1 release - THANKS: add names from 7.76.1 - misc: update copyright year ranges to match latest updates - [Tatsuhiro Tsujikawa brought this change] ngtcp2: Use ALPN h3-29 for now Fixes #6864 Cloes #6886 Jay Satiro (11 Apr 2021) - TODO: remove 18.22 --fail-with-body --fail-with-body was added in 8a964cb (precedes curl-7_76_0). Daniel Stenberg (10 Apr 2021) - [Jürgen Gmach brought this change] src/tool_vms.c: remove duplicated word in comment Closes #6881 - configure: fix CURL_DARWIN_CFLAGS use The macro name change was not completely done. Follow-up to 5d2c384452543c Bug: https://github.com/curl/curl/commit/5d2c384452543c7b6c9fb02eaa0afc84fd5ab941#commitcomment-49315187 Reported-by: Marcel Raad Closes #6878 - [Anthony Shaw brought this change] github/workflow: add "security-extended" to codeql-analysis.yml Extends the CodeQL code scan. Closes #6815 - [Jochem Broekhoff brought this change] examples/hiperfifo.c: check event_initialized before delete If event_del is called with the event struct (still) zeroed out, a segmentation fault may occur. event_initialized checks whether the event struct is nonzero. Closes #6876 - [Patrick Monnerat brought this change] ntlm: fix negotiated flags usage According to Microsoft document MS-NLMP, current flags usage is not accurate: flag NTLMFLAG_NEGOTIATE_NTLM2_KEY controls the use of extended security in an NTLM authentication message and NTLM version 2 cannot be negotiated within the protocol. The solution implemented here is: if the extended security flag is set, prefer using NTLM version 2 (as a server featuring extended security should also support version 2). If version 2 has been disabled at compile time, use extended security. Tests involving NTLM are adjusted to this new behavior. Fixes #6813 Closes #6849 - [Patrick Monnerat brought this change] ntlm: support version 2 on 32-bit platforms Closes #6849 - [Patrick Monnerat brought this change] curl_ntlm_core.h: simplify conditionals for USE_NTLM2SESSION ... as !defined(CURL_DISABLE_CRYPTO_AUTH) is a prerequisite for the whole NTLM. Closes #6849 - lib: remove unused HAVE_INET_NTOA_R* defines Closes #6867 - [Michael Forney brought this change] configure: include <time.h> unconditionally In 2682e5f5, several instances of AC_HEADER_TIME were removed since it is a deprecated autoconf macro. However, this was the macro that defined TIME_WITH_SYS_TIME, which was used to indicate that <time.h> can be included alongside <sys/time.h>. TIME_WITH_SYS_TIME is still used in the configure test body and since it is no longer defined, <time.h> is *not* included on systems that have <sys/time.h>. In particular, at least on musl libc and glibc, <sys/time.h> does not implicitly include <time.h> and does not declare clock_gettime, gmtime_r, or localtime_r. This causes configure to fail to detect those functions. The AC_HEADER_TIME macro deprecation text says > All current systems provide time.h; it need not be checked for. > Not all systems provide sys/time.h, but those that do, all allow > you to include it and time.h simultaneously. So, to fix this issue, simply include <time.h> unconditionally when testing for time-related functions and in libcurl, and don't bother checking for it. Closes #6859 - [Michael Forney brought this change] configure: remove use of RETSIGTYPE This was previously defined by the obsolete AC_TYPE_SIGNAL macro, which was removed in 2682e5f5. The deprecation text says > Your code may safely assume C89 semantics that RETSIGTYPE is void. So, remove it and just use void instead. Closes #6861 - [Muhammed Yavuz Nuzumlalı brought this change] install: add instructions for Apple Darwin platforms Closes #6860 - [Muhammed Yavuz Nuzumlalı brought this change] configure: disable min version set for Darwin Fixes #6838 Closes #6860 - [David Hu brought this change] docs/HTTP3.md: update the build instruction using gnutls In ngtcp2 the `with-gnutls` option is disabled by default, which will cause `curl` unable to be `make` because of lacking the libraries needed. Closes #6857 - RELEASE-NOTES: synced - typecheck-gcc: make the ssl-ctx-cb check use SSL_CTX pointers ... and not values. Reported-by: locpyl-tidnyd on github Fixes #6818 Closes #6819 - ngtcp2+gnutls: clear credentials when freed ... to avoid double-free. Reported-by: Kenneth Davidson Fixes #6824 Closes #6856 Jay Satiro (5 Apr 2021) - [Cherish98 brought this change] tool_progress: Fix progress meter in parallel mode Make sure the total amount of DL/UL bytes are counted before the transfer finalizes. Otherwise if a transfer finishes too quick, its total numbers are not added, and results in a DL%/UL% that goes above 100%. Detail: progress_meter() is called periodically, and it may not catch a transfer's total bytes if the value was unknown during the last call, and the transfer is finished and deleted (i.e., lost) during the next call. Closes https://github.com/curl/curl/pull/6840 - [Emil Engler brought this change] libssh: get rid of PATH_MAX This removes the last occurrence of PATH_MAX inside our libssh implementation by calculating the path length from the string length of the two components. Closes #6829 Daniel Stenberg (5 Apr 2021) - http_proxy: only loop on 407 + close if we have credentials ... to fix the retry-loop. Add test 718 to verify. Reported-by: Daniel Kurečka Fixes #6828 Closes #6850 - h2: allow 100 streams by default instead of 13, before the server has told how many streams it accepts. The server can always reject new streams anyway if we go above what it accepts. Ref: #6826 Closes #6852 - [Luke Granger-Brown brought this change] file: support GETing directories again After 957bc1881e686f9714c4e6a01bf33535091f0e21, we no longer compute an expected_size for directories. This has the upshot that when we compare even an empty Range with the available size, we fail. This brings back the previous behaviour, which was to succeed, but with empty content. This also removes the "Accept-ranges: bytes" header, which is nonsensical on directories. Adds test 3016 Fixes #6845 Closes #6846 - RELEASE-NOTES: synced and bumped to 7.76.1 - TLS: fix HTTP/2 selection for GnuTLS, BearSSL, mbedTLS, NSS, SChannnel, Secure Transport and wolfSSL... Regression since 88dd1a8a115b1f5ece (shipped in 7.76.0) Reported-by: Kenneth Davidson Reported-by: romamik om github Fixes #6825 Closes #6827 Jay Satiro (2 Apr 2021) - hostip: Fix for builds that disable all asynchronous DNS - Define Curl_resolver_error function only when USE_CURL_ASYNC. Prior to this change building curl without an asynchronous resolver backend (c-ares or threaded) and without DoH (DNS-over-HTTPS, which is also asynchronous but independent of resolver backend) would cause a build error since Curl_resolver_error is called by and evaluates variables only available in asynchronous builds. Reported-by: Benbuck Nason Fixes https://github.com/curl/curl/issues/6831 Closes https://github.com/curl/curl/pull/6832 Daniel Stenberg (31 Mar 2021) - [Gilles Vollant brought this change] openssl: Fix CURLOPT_SSLCERT_BLOB without CURLOPT_SSLCERT_KEY Reported-by: Christian Schmitz Fixes #6816 Closes #6820 Version 7.76.0 (31 Mar 2021) Daniel Stenberg (31 Mar 2021) - RELEASE-NOTES: synced curl 7.76.0 release - THANKS: added names from 7.76.0 - CURLOPT_AUTOREFERER.3: clarify that it sets the full URL ... some users may not want that! - define: remove CURL_DISABLE_NTLM ifdefs It was never defined anywhere. Fixed disable-scan (test 1165) to also scan headers, which found this issue. Closes #6809 - vtls: fix addsessionid for non-proxy builds Follow-up to b09c8ee15771c61 Fixes #6812 Closes #6811 - [Li Xinwei brought this change] cmake: support WinIDN Closes #6807 - transfer: clear 'referer' in declaration To silence (false positive) compiler warnings about it. Follow-up to 7214288898f5625 Reviewed-by: Marcel Raad Closes #6810 - [Marc Hoersken brought this change] config: fix SSPI enabling NTLM if crypto auth is disabled Avoid enabling NTLM feature based upon Windows SSPI being enabled in case that crypto auth is disabled. Reported-by: Marcel Raad Follow-up to #6277 Fixes #6803 Closes #6808 - HISTORY: add two 2021 events - vtls: add 'isproxy' argument to Curl_ssl_get/addsessionid() To make sure we set and extract the correct session. Reported-by: Mingtao Yang Bug: https://curl.se/docs/CVE-2021-22890.html CVE-2021-22890 - [Viktor Szakats brought this change] transfer: strip credentials from the auto-referer header field Added test 2081 to verify. CVE-2021-22876 Bug: https://curl.se/docs/CVE-2021-22876.html - curl_sasl: fix compiler error with --disable-crypto-auth ... if libgsasl was found. Closes #6806 - [Patrick Monnerat brought this change] ldap: only set the callback ptr for TLS context when TLS is used Follow-up to a5eee22e594c2460f Fixes #6804 Closes #6805 - copyright: update copyright year ranges to 2021 Reviewed-by: Emil Engler Closes #6802 - send_speed: simplify the checks for if a speed limit is set ... as we know the value cannot be set to negative: enforced by setopt() - http: cap body data amount during send speed limiting By making sure never to send off more than the allowed number of bytes per second the speed limit logic is given more room to actually work. Reported-by: Fabian Keil Bug: https://curl.se/mail/lib-2021-03/0042.html Closes #6797 - urldata: merge "struct DynamicStatic" into "struct UrlState" Both were used for the same purposes and there was no logical separation between them. Combined, this also saves 16 bytes in less holes in my test build. Closes #6798 - tests/README.md: mentioned that en_US.UTF-8 is required Reported-by: Oumph on github Fixes #6768 - HISTORY: fixed the Mac OS X 10.1 release date Based on what Wikipedia says Jay Satiro (26 Mar 2021) - examples: Remove threaded-shared-conn.c due to bug Known bug 11.11 is the shared object's connection cache is not thread safe, so we should not have an example for it. Ref: https://github.com/curl/curl/issues/4915 Ref: https://curl.se/docs/knownbugs.html#A_shared_connection_cache_is_not Closes https://github.com/curl/curl/pull/6795 - KNOWN_BUGS: Update 11.9 - DoH option inheritance - Add description: Explain that some options aren't inherited because they are not relevant for the DoH SSL connections or may result in unexpected behavior. - Remove the reference to #4578 (SSL verify options not inherited) since that was fixed by #6597 (separate DoH-specific options for verify). - Explain that DoH-specific options (those created by #6597) are available: CURLOPT_DOH_SSL_VERIFYHOST, CURLOPT_DOH_SSL_VERIFYPEER and CURLOPT_DOH_SSL_VERIFYSTATUS. - Add a reference to #6605 and explain that the user's debug function is not inherited because it would be unexpected to pass internal handles (ie DoH handles) to the user's callback. Closes https://github.com/curl/curl/issues/6605 Daniel Stenberg (26 Mar 2021) - curl_easy_setopt.3: add curl_easy_option* functions to SEE ALSO - [Jean-Philippe Menil brought this change] openssl: ensure to check SSL_CTX_set_alpn_protos return values SSL_CTX_set_alpn_protos() return 0 on success, and non-0 on failure Signed-off-by: Jean-Philippe Menil <jpmenil@gmail.com> Closes #6794 - multi: close the connection when h2=>h1 downgrading Otherwise libcurl is likely to reuse the connection again in the next attempt since the connection reuse logic doesn't take downgrades into account. Reported-by: Anthony Ramine Fixes #6788 Closes #6793 - openssl: set the transfer pointer for logging early Otherwise, the transfer will be NULL in the trace function when the early handshake details arrive and then curl won't show them. Regresssion in 7.75.0 Reported-by: David Hu Fixes #6783 Closes #6792 - RELEASE-NOTES: synced - TODO: Custom progress meter update interval Ref: https://stackoverflow.com/q/66789977/93747 - docs/ABI: tighten up the language Make the promises more firm Closes #6786 - openldap: disconnect better Instead of clearing the callback argument in disconnect, set it to the (new) transfer to make sure the correct data is passed to the callbacks. Follow-up to e467ea3bd937f38 Assisted-by: Patrick Monnerat Closes #6787 - libssh2: kdb_callback: get the right struct pointer After the recent conn/data refactor in this source file, this function was mistakenly still getting the old struct pointer which would lead to crash on servers with keyboard-interactive auth enabled. Follow-up to a304051620b92e12b (shipped in 7.75.0) Reported-by: Christian Schmitz Fixes #6691 Closes #6782 - tftp: remove unused struct fields Follow-up to d3d90ad9c00530d Closes #6781 - openldap: avoid NULL pointer dereferences Follow-up to a59c33ceffb8f78 Reported-by: Patrick Monnerat Fixes #6676 Closes #6780 - http: strip default port from URL sent to proxy To make sure the Host: header and the URL provide the same authority portion when sent to the proxy, strip the default port number from the URL if one was provided. Reported-by: Michael Brown Fixes #6769 Closes #6778 - azure: disable test 433 on azure-ubuntu Something in that environment sets XDG_CONFIG_HOME for us in a way that breaks the test. Reported-by: Marc Hörsken Fixes #6739 Closes #6777 - tftp: remove the 3600 second default timeout ... it was never meant to be there. Reported-by: Tomas Berger Fixes #6774 Closes #6776 - docs: make gen.pl support *italic* and **bold** Remove some nroffisms from the cmdline doc files to simplify editing, and instead support this markdown style. Closes #6771 - ngtcp2: sync with recent API updates Closes #6770 - RELEASE-NOTES: synced - libssh2:ssh_connect: clear session pointer after free If libssh2_knownhost_init() returns NULL, like in an OOM situation, the ssh session was freed but the pointer wasn't cleared which made libcurl later call libssh2 to cleanup using the stale pointer. Fixes #6764 Closes #6766 - [Jacob Hoffman-Andrews brought this change] docs: document version of crustls dependency This also pins a specific release in the Travis test so future API-breaking changins in crustls won't break curl builds. Add RUSTLS documentation to release tarball. Enable running tests for rustls, minus FTP tests (require connect_blocking, which rustls doesn't implement) and 313 (requires CRL handling). Closes #6763 - [Jacob Hoffman-Andrews brought this change] rustls: Handle close_notify. If we get a close_notify, treat that as EOF. If we get an EOF from the TCP stream, treat that as an error (because we should have ended the connection earlier, when we got a close_notify). Closes #6763 - docs: clarify timeouts for queued transfers in multi API Closes #6758 - ftpserver: only load the preprocessed test file We always preprocess and tests are no longer sensible to load "raw" Closes #6738 - tests: use %TESTNUMBER instead of fixed number This makes the tests easier to copy and relocate to other test numbers without having to update content. Closes #6738 - KNOWN_BUGS: CURLOPT_OPENSOCKETPAIRFUNCTION is missing Closes #5747 - TODO: provide timing info for each redirect Closes #6743 Jay Satiro (17 Mar 2021) - docs: Add SSL backend names to CURL_SSL_BACKEND - Document the names that can be used with CURL_SSL_BACKEND: bearssl, gnutls, gskit, mbedtls, mesalink, nss, openssl, rustls, schannel, secure-transport, wolfssl Ref: https://github.com/curl/curl/issues/2209#issuecomment-360623286 Ref: https://github.com/curl/curl/issues/6717#issuecomment-800745201 Closes https://github.com/curl/curl/pull/6755 - docs: Explain DOH transfers inherit some SSL settings - Document in DOH that some SSL settings are inherited but DOH hostname and peer verification are not and are controlled separately. - Document that CURLOPT_SSL_CTX_FUNCTION is inherited by DOH handles but we're considering changing behavior to no longer inherit it. Request feedback. Closes https://github.com/curl/curl/pull/6688 Daniel Stenberg (17 Mar 2021) - http: make 416 not fail with resume + CURLOPT_FAILONERRROR When asked to resume a download, libcurl will convert that to HTTP logic and if then the entire file is already transferred it will result in a 416 response from the HTTP server. With CURLOPT_FAILONERRROR set in that scenario, it should *not* lead to an error return. Updated test 1156, added test 1273 Reported-by: Jonathan Watt Fixes #6740 Closes #6753 - Curl_timeleft: check both timeouts during connect The duration of a connect and the total transfer are calculated from two different time-stamps. It can end up with the total timeout triggering before the connect timeout expires and we should make sure to acknowledge whichever timeout that is reached first. This is especially notable when a transfer first sits in PENDING, as that time is counted in the total time but the connect timeout is based on the time since the handle changed to the CONNECT state. The CONNECTTIMEOUT is per connect attempt. The TIMEOUT is for the entire operation. Fixes #6744 Closes #6745 Reported-by: Andrei Bica Assisted-by: Jay Satiro - configure: remove use of deprecated macros AC_HEADER_TIME, AC_HEADER_STDC and AC_TYPE_SIGNAL - configure: make AC_TRY_* into AC_*_IFELSE ... as the former versions are deprecated. - configure: s/AC_HELP_STRING/AS_HELP_STRING AC_HELP_STRING is deprecated in 2.70+ and I believe AS_HELP_STRING works already since 2.59 so bump the minimum required version to that. Reported-by: Emil Engler Fixes #6647 Closes #6748 - RELEASE-NOTES: synced - travis: use ubuntu nghttp2 package instead of build our own Closes #6751 - travis: bump wolfssl to 4.7.0 - travis: only build wolfssl when needed Closes #6751 - [Jacob Hoffman-Andrews brought this change] rustls: allocate a buffer for TLS data. Previously, rustls was using an on-stack array for TLS data. However, crustls has an (unusual) requirement that buffers it deals with are initialized before writing to them. By using calloc, we can ensure the buffer is initialized once and then reuse it across calls. Closes #6742 - travis: add a rustls build ... that doesn't run any tests (yet) Closes #6750 - HTTP2: remove the outdated remark about multiplexing for the tool - [Robert Ronto brought this change] http2: don't set KEEP_SEND when there's no more data to be sent this should fix an issue where curl sometimes doesn't send out a request with authorization info after a 401 is received over http2 Closes #6747 Marc Hoersken (15 Mar 2021) - config: fix building SMB with configure using Win32 Crypto Align conditions for NTLM features between CMake and configure builds by differentiating between USE_NTLM and USE_CURL_NTLM_CORE, just like curl_setup.h does internally to detect support of: - USE_NTLM: required for NTLM crypto authentication feature - USE_CURL_NTLM_CORE: required for SMB protocol Implement USE_WIN32_CRYPTO detection by checking for Crypt functions in wincrypt.h which are not available in the Windows App environment. Link advapi32 and crypt32 for Crypto API and Schannel SSL backend. Fix condition of Schannel SSL backend in CMake build accordingly. Reviewed-by: Marcel Raad Closes #6277 - config: fix detection of restricted Windows App environment Move the detection of the restricted Windows App environment in curl_setup.h before the definition of USE_WIN32_CRYPTO via included config-win32.h in case no build system is used. Reviewed-by: Marcel Raad Part of #6277 Daniel Stenberg (15 Mar 2021) - HISTORY: curl 7.7.2 was the first version used in Mac OS X 10.1 - gen.pl: quote "bare" minuses in the nroff curl.1 Reported-by: Alejandro Colomar Fixes #6698 Closes #6722 Daniel Gustafsson (14 Mar 2021) - hsts: remove unused defines MAX_HSTS_SUBLEN and MAX_HSTS_SUBLENSTR were unused from the initial commit, and mostly likely leftovers from early development. Remove as they're not used for anything. Closes #6741 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (12 Mar 2021) - github: add torture-ftp for FTP-only torture testing and at 20% to try to keep the run-time reasonable Closes #6728 - travis: split "torture" into a separate "events" build as well Run torture without FTP and reducing coverage to 20% For some reason the torture tests now run a lot slower on travis and run into the 50 minute limit all the time. Closes #6728 - ftp: fix memory leak in ftp_done If after a transfer is complete Curl_GetFTPResponse() returns an error, curl would not free the ftp->pathalloc block. Found by torture-testing test 576 Closes #6737 - [oxalica brought this change] http2: fail if connection terminated without END_STREAM Closes #6736 - RELEASE-NOTES: synced - [Jacob Hoffman-Andrews brought this change] rustls: support CURLOPT_SSL_VERIFYPEER This requires the latest main branch of crustls, which provides rustls_client_config_builder_dangerous_set_certificate_verifier and rustls_client_config_builder_set_enable_sni. This refactors the session setup into its own function, and adds a new function cr_hostname_is_ip. Because crustls doesn't support verification of IP addresses, special handling is needed: We disable SNI and set a placeholder hostname (which never actually gets sent on the wire). Closes #6719 Daniel Gustafsson (12 Mar 2021) - cookies: Fix potential NULL pointer deref with PSL Curl_cookie_init can be called with data being NULL, and this can in turn be passed to Curl_cookie_add, meaning that both functions must be careful to only use data where it's checked for being a NULL pointer. The libpsl support code does however dereference data without checking, so if we are indeed having an unset data pointer we cannot PSL check the cookiedomain. This is currently not a reachable dereference, as the only caller with a NULL data isn't passing a file to initialize cookies from, but since the API has this contract let's ensure we hold it. Closes #6731 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (12 Mar 2021) - [Michael Hordijk brought this change] configure: only add OpenSSL paths if they are defined Add paths for OpenSSL compiling and linking only if they have been defined. If they haven't been defined, we'll assume that the paths are already available to the toolchain. Closes #6730 Jay Satiro (12 Mar 2021) - retry.d: Clarify transient 5xx HTTP response codes - Clarify the only 5xx response codes that are treated as transient are 500, 502, 503 and 504. Prior to this change it said it treated all 5xx as transient, but the code says otherwise. Ref: https://github.com/curl/curl/blob/curl-7_75_0/src/tool_operate.c#L462-L495 Closes https://github.com/curl/curl/pull/6724 - retry-all-errors.d: Explain curl errors versus HTTP response errors - Add a paragraph explaining that curl does not consider HTTP response errors as curl errors, and how that behavior can be modified by using --retry and --fail. The --retry-all-errors doc says "Retry on any error" which some users may find misleading without the added explanation. Ref: https://curl.se/docs/faq.html#Why_do_I_get_downloaded_data_eve Ref: https://curl.se/docs/faq.html#curl_doesn_t_return_error_for_HT Reported-by: Lawrence Gripper Fixes https://github.com/curl/curl/issues/6712 Closes https://github.com/curl/curl/pull/6720 Daniel Stenberg (11 Mar 2021) - travis: switch ngtcp2 build over to quictls The ngtcp2 project switched over to using the quictls OpenSSL fork instead of their own patched OpenSSL. We follow suit. Closes #6729 - test220/314: adjust to run with Hyper - c-hyper: support automatic content-encoding Closes #6727 - http: remove superfluous NULL assign Closes #6727 - tool_operate: bail if set CURLOPT_HTTP09_ALLOWED returns error Closes #6727 - setopt: error on CURLOPT_HTTP09_ALLOWED set true with Hyper Not supported. Closes #6727 - test306: make it not run with Hyper ... as it tests HTTP/0.9 which Hyper doesn't support. - test304: header CRLF cleanup to work with Hyper - FTP: allow SIZE to fail when doing (resumed) upload Added test 362 to verify. Reported-by: Jordan Brown Regression since 7ea2e1d0c5a7f (7.73.0) Fixes #6715 Closes #6725 - configure: provide Largefile feature for curl-config ... as cmake now does it correctly, and make test1014 check for it Closes #6702 - config: remove CURL_SIZEOF_CURL_OFF_T use only SIZEOF_CURL_OFF_T Make the code consistently use a single name for the size of the "curl_off_t" type. Closes #6702 Jay Satiro (10 Mar 2021) - [Jun-ya Kato brought this change] ngtcp2: Fix build error due to change in ngtcp2_addr_init ngtcp2/ngtcp2@b8d90a9 changed the function prototype. Closes https://github.com/curl/curl/pull/6716 Daniel Stenberg (10 Mar 2021) - [ejanchivdorj brought this change] multi: update pending list when removing handle when removing a handle, most of the lists are updated but pending list is not updated. Updating now. Closes #6713 - [kokke brought this change] lib1536: check ptr against NULL before dereferencing it Closes #6710 - [kokke brought this change] lib1537: check ptr against NULL before dereferencing it Fixes #6707 Closes #6708 - travis: make torture tests skip TLS-SRP tests ... as it seems to often hang. Also: skip the "normal" tests as they're already run by many other builds. Closes #6705 - openssl: adapt to v3's new const for a few API calls Closes #6703 - quiche: fix crash when failing to connect Reported-by: ウさん Fixes #6664 Closes #6701 - RELEASE-NOTES: synced Fixed the release counter and added a missing contributor - RELEASE-NOTES: synced - dynbuf: bump the max HTTP request to 1MB Raised from 128KB to allow longer request headers. Reported-by: Carl Zogheib Fixes #6681 Closes #6685 Jay Satiro (6 Mar 2021) - schannel: Evaluate CURLOPT_SSL_OPTIONS via SSL_SET_OPTION macro - Change use of those options from CURLOPT_SSL_OPTIONS that are not already evaluated via SSL_SET_OPTION in schannel and secure transport to use that instead of data->set.ssl.optname. Example: Evaluate SSL_SET_OPTION(no_revoke) instead of data->set.ssl.no_revoke. This change is because options set via CURLOPT_SSL_OPTIONS (data->set.ssl.optname) are separate from those set for HTTPS proxy via CURLOPT_PROXY_SSL_OPTIONS (data->set.proxy_ssl.optname). The SSL_SET_OPTION macro determines whether the connection is for HTTPS proxy and based on that which option to evaluate. Since neither Schannel nor Secure Transport backends currently support HTTPS proxy in libcurl, this change is for posterity and has no other effect. Closes https://github.com/curl/curl/pull/6690 - [kokke brought this change] c-hyper: Remove superfluous pointer check `n` pointer is never NULL once set. Found by static analysis. Ref: https://github.com/curl/curl/issues/6696 Closes https://github.com/curl/curl/pull/6697 - version.d: Add missing features to the features list - Add missing entries for gsasl, Kerberos, NTLM_WB, TrackMemory, Unicode and zstd. - Remove krb4 since it's no longer a feature. Reported-by: Ádler Jonas Gross Fixes https://github.com/curl/curl/issues/6677 Closes https://github.com/curl/curl/pull/6687 - [Vladimir Varlamov brought this change] docs: add missing Arg tag to --stderr Prior to this change the required argument was not shown. curl.1 before: --stderr curl.1 after: --stderr <file> curl --help before: --stderr Where to redirect stderr curl --help after: --stderr <file> Where to redirect stderr Closes https://github.com/curl/curl/pull/6692 - projects: Update VS projects for OpenSSL 1.1.x - Update VS project templates to use the OpenSSL lib names and include directories for OpenSSL 1.1.x. This change means the VS project files will now build only with OpenSSL 1.1.x when an OpenSSL configuration is chosen. Prior to this change the project files built only with OpenSSL 1.0.x (end-of-life) when an OpenSSL configuration was chosen. The template changes in this commit were made by script: libeay32.lib => libcrypto.lib ssleay32.lib => libssl.lib ..\..\..\..\..\openssl\inc32 => ..\..\..\..\..\openssl\include And since the output directory now contains the includes it's prepended: ..\..\..\..\..\openssl\build\Win{32,64}\VC{6..15}\{DLL,LIB} {Debug,Release}\include - Change build-openssl.bat to copy the build's include directory to the output directory (as seen above). Each build has its own opensslconf.h which is different so we can't just include the source include directory any longer. Note the include directory in the output directory is a full copy from the build so technically we don't need to include the OpenSSL source include directory in the template. However, I left it last in case the user made a custom OpenSSL build using the old method which would put opensslconf in the OpenSSL source include directory. - Change build-openssl.bat to use a temporary install directory that is different from the temporary build directory. For OpenSSL 1.1.x the temporary paths must be separate not a descendant of the other, otherwise pdb files will be lost between builds. Ref: https://curl.se/mail/lib-2018-10/0049.html Ref: https://gist.github.com/jay/125191c35bbeb894444eff827651f755 Ref; https://github.com/openssl/openssl/issues/10005 Fixes https://github.com/curl/curl/issues/984 Closes https://github.com/curl/curl/pull/6675 - doh: Inherit CURLOPT_STDERR from user's easy handle Prior to this change if the user set their easy handle's error stream to something other than stderr it was not inherited by the doh handles, which meant that they would still write to the default standard error stream (stderr) for verbose output. Bug: https://github.com/curl/curl/issues/6605 Reported-by: arvids-kokins-bidstack@users.noreply.github.com Closes https://github.com/curl/curl/pull/6661 Marc Hoersken (1 Mar 2021) - CI/azure: replace python-impacket with python3-impacket As of this month Azure DevOps uses Ubuntu 20.04 LTS which no longer supports Python 2 and instead ships Python 3. Closes #6678 - runtests.pl: kill processes locking test log files Introduce a new runtests.pl command option: -rm For now only required and implemented for Windows. Ignore stunnel logs due to long running processes. Requires Sysinternals handle[64].exe to be on PATH. Reviewed-by: Jay Satiro Ref: #6058 Closes #6179 - pathhelp.pm: fix use of pwd -L in Msys environment While Msys2 has a pwd binary which supports -L, Msys1 only has a shell built-in with that feature. Reviewed-by: Jay Satiro Part of #6179 Daniel Gustafsson (1 Mar 2021) - ldap: use correct memory free function unescaped is coming from Curl_urldecode and not a unicode conversion function, so reclaiming its memory should be performed with a normal call to free rather than curlx_unicodefree. In reality, this is the same thing as curlx_unicodefree is implemented as a call to free but that's not guaranteed to always hold. Using the curlx macro present issues with memory debugging as well. Closes #6671 Reviewed-by: Jay Satiro <raysatiro@yahoo.com> Reviewed-by: Daniel Stenberg <daniel@haxx.se> - url: fix typo in comment Correct a small typo which snuck in with a304051620. Jay Satiro (28 Feb 2021) - tool_help: Increase space between option and description - Increase the minimum number of spaces between the option and the description from 1 to 2. Before: ~~~ -u, --user <user:password> Server user and password -A, --user-agent <name> Send User-Agent <name> to server -v, --verbose Make the operation more talkative -V, --version Show version number and quit -w, --write-out <format> Use output FORMAT after completion --xattr Store metadata in extended file attributes ~~~ After: ~~~ -u, --user <user:password> Server user and password -A, --user-agent <name> Send User-Agent <name> to server -v, --verbose Make the operation more talkative -V, --version Show version number and quit -w, --write-out <format> Use output FORMAT after completion --xattr Store metadata in extended file attributes ~~~ Closes https://github.com/curl/curl/pull/6674 Daniel Stenberg (27 Feb 2021) - curl: set CURLOPT_NEW_FILE_PERMS if requested The --create-file-mode code logic accepted the value but never actually passed it on to libcurl! Follow-up to a7696c73436f (shipped in 7.75.0) Reported-by: Johannes Lesr Fixes #6657 Closes #6666 - tool_operate: check argc before accessing argv[1] Follow-up to 09363500b Reported-by: Emil Engler Reviewed-by: Daniel Gustafsson Closes #6668 Daniel Gustafsson (26 Feb 2021) - [Jean-Philippe Menil brought this change] openssl: remove get_ssl_version_txt in favor of SSL_get_version openssl: use SSL_get_version to get connection protocol Replace our bespoke get_ssl_version_txt in favor of SSL_get_version. We can get rid of few lines of code, since SSL_get_version achieve the exact same thing Closes #6665 Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Signed-off-by: Jean-Philippe Menil <jpmenil@gmail.com> - gnutls: Fix nettle discovery Commit e06fa7462ac258c removed support for libgcrypt leaving only support for nettle which has been the default crypto library in GnuTLS for a long time. There were however a few conditionals on USE_GNUTLS_NETTLE which cause compilation errors in the metalink code (as it used the gcrypt fallback instead as a result). See the below autobuild for an example of the error: https://curl.se/dev/log.cgi?id=20210225123226-30704#prob1 This removes all uses of USE_GNUTLS_NETTLE and also removes the gcrypt support from the metalink code while at it. Closes #6656 Reviewed-by: Daniel Stenberg <daniel@haxx.se> - cookies: Support multiple -b parameters Previously only a single -b cookie parameter was supported with the last one winning. This adds support for supplying multiple -b params to have them serialized semicolon separated. Both cookiefiles and cookies can be entered multiple times. Closes #6649 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (25 Feb 2021) - build: remove all traces of USE_BLOCKING_SOCKETS libcurl doesn't behave properly with the define set Closes #6655 - RELEASE-NOTES: synced Daniel Gustafsson (25 Feb 2021) - docs: Fix typos Random typos spotted when skimming docs. - cookies: Use named parameters in header prototypes Align header with project style of using named parameters in the function prototypes to aid readability and self-documentation. Closes #6653 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (24 Feb 2021) - urldata: make 'actions[]' use unsigned char instead of int ... as it only needs a few bits per index anyway. Reviewed-by: Daniel Gustafsson Closes #6648 - configure: fail if --with-quiche is used and quiche isn't found Closes #6652 - [Gregor Jasny brought this change] cmake: use CMAKE_INSTALL_INCLUDEDIR indirection Reviewed-by: Sergei Nikulov Closes #6440 Viktor Szakats (23 Feb 2021) - mingw: enable using strcasecmp() This makes the 'Features:' list sorted case-insensitively, bringing output in-line with *nix builds. Reviewed-by: Jay Satiro Closes #6644 - build: delete unused feature guards - `HAVE_STRNCASECMP` - `HAVE_TCGETATTR` - `HAVE_TCSETATTR` Reviewed-by: Jay Satiro Reviewed-by: Daniel Stenberg Closes #6645 Jay Satiro (23 Feb 2021) - docs: add CURLOPT_CURLU to 'See also' in curl_url_ functions Closes https://github.com/curl/curl/pull/6639 Daniel Stenberg (23 Feb 2021) - [Jacob Hoffman-Andrews brought this change] configure: make hyper opt-in, and fail if missing Previously, configure would look for hyper by default, and use it if found; otherwise it would not use hyper, and not error. Now, configure will not look for hyper unless --with-hyper is passed. If configure looks for hyper and fails, it will error. Also, add -ld -lpthread -lm to Hyper's libs. I think they are required. Closes #6598 - multi: do once-per-transfer inits in before_perform in DID state ... since the state machine might go to RATELIMITING and then back to PERFORMING doing once-per-transfer inits in that function is wrong and it caused problems with receiving chunked HTTP and it set the PRETRANSFER time much too often... Regression from b68dc34af341805aeb7b3715 (shipped in 7.75.0) Reported-by: Amaury Denoyelle Fixes #6640 Closes #6641 - RELEASE-NOTES: synced - CODE_STYLE.md: fix broken link to INTERNALS ... the link would only work if browsed on GitHub, while this link now takes the user to the website instead and thus should work on either. Reported-by: David Demelier - curl_url_set.3: mention CURLU_PATH_AS_IS ... it has been supported since the URL API was added. Bug: https://curl.se/mail/lib-2021-02/0046.html Closes #6638 Viktor Szakats (21 Feb 2021) - time: enable 64-bit time_t in supported mingw environments (Unless 32-bit `time_t` is selected manually via the `_USE_32BIT_TIME_T` mingw macro.) Previously, 64-bit `time_t` was enabled on VS2005 and newer only, and 32-bit `time_t` was used on all other Windows builds. Assisted-by: Jay Satiro Closes #6636 Jay Satiro (20 Feb 2021) - test1188: Check for --fail HTTP status - Change the test to check for curl error on HTTP 404 Not Found. test1188 tests "--write-out with %{onerror} and %{urlnum} to stderr". Prior to this change it did that by specifying a non-existent host which would cause an error. ISPs may hijack DNS and resolve non-existent hosts so the test would not work if that was the case. Ref: https://en.wikipedia.org/wiki/DNS_hijacking#Manipulation_by_ISPs Ref: https://github.com/curl/curl/issues/6621 Ref: https://github.com/curl/curl/pull/6623 Closes https://github.com/curl/curl/pull/6637 - memdebug: close debug logfile explicitly on exit - Use atexit to register a dbg cleanup function that closes the logfile. LeakSantizier (LSAN) calls _exit() instead of exit() when a leak is detected on exit so the logfile must be closed explicitly or data could be lost. Though _exit() does not call atexit handlers such as this, LSAN's call to _exit() comes after the atexit handlers are called. Prior to this change the logfile was not explicitly closed so it was possible that if LSAN detected a leak and called _exit (which does not flush or close files like exit) then the logfile could be missing data. That could then cause curl's memanalyze to report false leaks (eg a malloc was recorded to the logfile but the corresponding free was discarded from the buffer instead of written to the logfile, then memanalyze reports that as a leak). Ref: https://github.com/google/sanitizers/issues/1374 Bug: https://github.com/curl/curl/pull/6591#issuecomment-780396541 Closes https://github.com/curl/curl/pull/6620 - curl_multibyte: always return a heap-allocated copy of string - Change the Windows char <-> UTF-8 conversion functions to return an allocated copy of the passed in string instead of the original. Prior to this change the curlx_convert_ functions would, as what I assume was an optimization, not make a copy of the passed in string if no conversion was required. No conversion is required in non-UNICODE Windows builds since our tchar strings are type char and remain in whatever the passed in encoding is, which is assumed to be UTF-8 but may be other encoding. In contrast the UNICODE Windows builds require conversion (wchar <-> char) and do return a copy. That inconsistency could lead to programming errors where the developer expects a copy, and does not realize that won't happen in all cases. Closes https://github.com/curl/curl/pull/6602 Viktor Szakats (19 Feb 2021) - http: add new files missed from referrer commit Ref: 44872aefc2d54f297caf2b0cc887df321bc9d791 Ref: #6591 - http: add support to read and store the referrer header - add CURLINFO_REFERER libcurl option - add --write-out '%{referer}' command-line option - extend --xattr command-line option to fill user.xdg.referrer.url extended attribute with the referrer (if there was any) Closes #6591 Daniel Stenberg (19 Feb 2021) - urldata: remove the _ORIG suffix from string names It doesn't provide any useful info but only makes the names longer. Closes #6624 - url: fix memory leak if OOM in the HSTS handling Reported-by: Viktor Szakats Bug: https://github.com/curl/curl/pull/6627#issuecomment-781626205 Closes #6628 - gnutls: assume nettle crypto support nettle has been the default crypto library with GnuTLS since 2010. By dropping support for the previous libcrypto, we simplify code. Closes #6625 - asyn-ares: use consistent resolve error message ... with the help of Curl_resolver_error() which now is moved from asyn-thead.c and is provided globally for this purpose. Follow-up to 35ca04ce1b77636 Makes test 1188 work for c-ares builds Closes #6626 Viktor Szakats (18 Feb 2021) - ci: stop building on freebsd-12-1 An updated freebsd-12-2 image was added a few months ago, and this older one is consistently failing to go past `pkginstall`: ``` Newer FreeBSD version for package py37-mlt: To ignore this error set IGNORE_OSVERSION=yes - package: 1202000 - running kernel: 1201000 Ignore the mismatch and continue? [Y/n]: pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:12:amd64 ``` FreeBSD thread suggests that 12.1 is EOL, and best to avoid. Ref: https://forums.freebsd.org/threads/78856/ Reviewed-by: Daniel Stenberg Closes #6622 Daniel Stenberg (18 Feb 2021) - test1188: change error from connect to resolve error Using the %NOLISTENPORT to trigger a connection failure is somewhat "risky" (since it isn't guaranteed to not be listened to) and caused occasional CI problems. This fix changes the infused error to be a more reliable one but still verifies the --write-out functionality properly - which is the purpose of this test. Reported-by: Jay Satiro Fixes #6621 Closes #6623 - url.c: use consistent error message for failed resolve - BUGS: language polish - wolfssl: don't store a NULL sessionid This caused a memory leak as the session id cache entry was still erroneously stored with a NULL sessionid and that would later be treated as not needed to get freed. Reported-by: Gisle Vanem Fixes #6616 Closes #6617 - parse_proxy: fix a memory leak in the OOM path Reported-by: Jay Satiro Reviewed-by: Jay Satiro Reviewed-by: Emil Engler Closes #6614 Bug: https://github.com/curl/curl/pull/6591#issuecomment-780396541 Jay Satiro (17 Feb 2021) - url: fix possible use-after-free in default protocol Prior to this change if the user specified a default protocol and a separately allocated non-absolute URL was used then it was freed prematurely, before it was then used to make the replacement URL. Bug: https://github.com/curl/curl/issues/6604#issuecomment-780138219 Reported-by: arvids-kokins-bidstack@users.noreply.github.com Closes https://github.com/curl/curl/pull/6613 Daniel Stenberg (16 Feb 2021) - multi: rename the multi transfer states While working on documenting the states it dawned on me that step one is to use more descriptive names on the states. This also changes prefix on the states to make them shorter in the source. State names NOT ending with *ing are transitional ones. Closes #6612 Viktor Szakats (16 Feb 2021) - http: do not add a referrer header with empty value Previously an empty 'Referer:' header was added to the HTTP request when passing `--referer ';auto'` or `--referer ''` on the command-line. This patch makes `--referer` work like `--header 'Referer:'` and will only add the header if it has a non-zero length value. Reviewed-by: Jay Satiro Closes #6610 Daniel Stenberg (16 Feb 2021) - lib: remove 'conn->data' completely The Curl_easy pointer struct entry in connectdata is now gone. Just before commit 215db086e0 landed on January 8, 2021 there were 919 references to conn->data. Closes #6608 - openldap: pass 'data' to the callbacks instead of 'conn' Jay Satiro (15 Feb 2021) - doh: Fix sharing user's resolve list with DOH handles - Share the shared object from the user's easy handle with the DOH handles. Prior to this change if the user had set a shared object with shared cached DNS (CURL_LOCK_DATA_DNS) for their easy handle then that wasn't used by any associated DOH handles, since they used the multi's default hostcache. This change means all the handles now use the same hostcache, which is either the shared hostcache from the user created shared object if it exists or if not then the multi's default hostcache. Reported-by: Manuj Bhatia Fixes https://github.com/curl/curl/issues/6589 Closes https://github.com/curl/curl/pull/6607 Daniel Stenberg (15 Feb 2021) - http2: remove conn->data use ... but instead use a private alternative that points to the "driving transfer" from the connection. We set the "user data" associated with the connection to be the connectdata struct, but when we drive transfers the code still needs to know the pointer to the transfer. We can change the user data to become the Curl_easy handle, but with older nghttp2 version we cannot dynamically update that pointer properly when different transfers are used over the same connection. Closes #6520 - openssl: remove conn->data use We still make the trace callback function get the connectdata struct passed to it, since the callback is anchored on the connection. Repeatedly updating the callback pointer to set 'data' with SSL_CTX_set_msg_callback_arg() doesn't seem to work, probably because there might already be messages in the queue with the old pointer. This code therefore makes sure to set the "logger" handle before using OpenSSL calls so that the right easy handle gets used for tracing. Closes #6522 - RELEASE-NOTES: synced Jay Satiro (14 Feb 2021) - doh: add options to disable ssl verification - New libcurl options CURLOPT_DOH_SSL_VERIFYHOST, CURLOPT_DOH_SSL_VERIFYPEER and CURLOPT_DOH_SSL_VERIFYSTATUS do the same as their respective counterparts. - New curl tool options --doh-insecure and --doh-cert-status do the same as their respective counterparts. Prior to this change DOH SSL certificate verification settings for verifyhost and verifypeer were supposed to be inherited respectively from CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER, but due to a bug were not. As a result DOH verification remained at the default, ie enabled, and it was not possible to disable. This commit changes behavior so that the DOH verification settings are independent and not inherited. Ref: https://github.com/curl/curl/pull/4579#issuecomment-554723676 Fixes https://github.com/curl/curl/issues/4578 Closes https://github.com/curl/curl/pull/6597 - hostip: fix crash in sync resolver builds that use DOH - Guard some Curl_async accesses with USE_CURL_ASYNC instead of !CURLRES_SYNCH. This is another follow-up to 8335c64 which moved the async struct from the connectdata struct into the Curl_easy struct. A previous follow-up 6cd167a fixed building for sync resolver by guarding some async struct accesses with !CURLRES_SYNCH. The problem is since DOH (DNS-over-HTTPS) is available as an asynchronous secondary resolver the async struct may be used even when libcurl is built for the sync