Check which eMudhra path a trust store still accepts
A subject search of the active Debian trust bundle still printed three emSign roots after the October security update. The same bundle rejected every published eMudhra intermediate. Those two facts are easy to collapse into one, and they are not the same fact.
The check below used Debian 13, ca-certificates 20250419+deb13u1, OpenSSL 3.5.7, curl 8.14.1, Python 3.13.5, and Node.js 22.23.4. curl to https://example.com/ returned HTTP 200. The same curl failed against the public eMudhra test hosts with ssl_verify_result 20, unable to get local issuer certificate. Node.js 22.23.4, including when started with --use-openssl-ca, failed those hosts with CERT_REVOKED instead. A successful connection to a known site and a failed connection to the test hosts is the result this review is looking for. A grep hit on the string emSign is not.
Mozilla reported on 20 October 2026 that private keys for eight eMudhra intermediate certificates had been compromised. The report reached Mozilla on 17 October. Firefox was protected through OneCRL the same day. NSS 3.131.1, released on 20 October, removes trust in three roots and says the removal of emSign ECC Root CA - G3 is the incident response. Debian’s security upload does a narrower thing. Its changelog says it blacklists emSign ECC Root CA - G3 and Comodo AAA Services root. It does not say that every remaining emSign certificate is part of that blacklist.
The G3 label is two certificates
The intermediates that signed the test-host certificates are not issued by emSign ECC Root CA - G3. The direct issuer on one path is emSign Root TLS CA - G3. That root was not in the Debian Mozilla certificate directory in ca-certificates 20250419, the package version before the security upload. The served chain reaches a trust anchor only because emSign Root TLS CA - G3 is cross-signed by emSign ECC Root CA - G3.
Those two subjects are not a naming variation of one file. The incident inventory gives them different SHA-256 fingerprints:
7DD78D5F4F13459A83DFF9ABBBA62EDBAF6F2D102BF257FD712F4D9F2746ED8D emSign Root TLS CA - G3
86A1ECBA089C4A8D3BBE2734C612BA341D813E043CF9E8A862CD5C57A36BBE6B emSign ECC Root CA - G3
D7A7A0FB5D7E2731D771E9484EBCDEF71D5F0C3E0A2948782BC83EE0EA699EF4 AAA Certificate Services
The fingerprint for emSign ECC Root CA - G3 matched the file in the pre-update Debian package. The fingerprint for AAA Certificate Services matched Comodo_AAA_Services_root.crt in that same package. emSign Root TLS CA - G3 was not a file in that package at all.
An intermediate issued by emSign Root TLS CA - G3 verified when that root was supplied alone as -CAfile. The same intermediate failed, with error 20, when emSign ECC Root CA - G3 was supplied alone. The names share G3. The signature check does not. A review that opens the ECC file because the issuer string contains G3 is looking at the cross-signer, not at the issuer of the intermediate.
The other public path is the Sectigo cross-sign. The same intermediate name, EM DV TLS CA - G3B, also exists as a certificate issued by AAA Certificate Services. That second certificate verified when the AAA root was supplied as -CAfile, and it failed when the ECC G3 root was supplied. Disabling one of those roots does not answer for the other path.
The incident write-up says the intermediate keys were generated incorrectly and can be recovered from the certificates. The chain check does not need that method, and this article does not repeat it. The useful question for a non-browser client is whether a path from the served chain still ends at a local trust anchor.
Read the blacklist, then the bundle
On the tested host the installed package was ca-certificates 20250419+deb13u1. The changelog entry for that upload is short:
ca-certificates (20250419+deb13u1) trixie-security; urgency=high
* Non-maintainer upload by the Security Team.
* Blacklist "emSign ECC Root CA - G3" and "Comodo AAA Services root"
(Closes: #1151413)
The active configuration showed the same decision as a leading ! on two lines, and no ! on three other emSign lines:
!mozilla/Comodo_AAA_Services_root.crt
mozilla/emSign_ECC_Root_CA_-_C3.crt
!mozilla/emSign_ECC_Root_CA_-_G3.crt
mozilla/emSign_Root_CA_-_C1.crt
mozilla/emSign_Root_CA_-_G1.crt
The line numbers on another host can differ. The ! is the marker update-ca-certificates uses for a certificate that remains known but is not trusted. After the update, emSign_ECC_Root_CA_-_G3.crt and Comodo_AAA_Services_root.crt were absent from /usr/share/ca-certificates/mozilla/. The three enabled files were still there.
Search the bundle the clients actually load, not only the configuration file. A disabled line that was never rebuilt into /etc/ssl/certs/ca-certificates.crt would be a different defect from a line that was rebuilt correctly.
openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-certificates.crt \
| openssl pkcs7 -print_certs -noout \
| grep -E 'emSign|AAA Certificate Services'
On this host the command printed three subjects and stopped:
subject=C=US, OU=emSign PKI, O=eMudhra Inc, CN=emSign ECC Root CA - C3
issuer=C=US, OU=emSign PKI, O=eMudhra Inc, CN=emSign ECC Root CA - C3
subject=C=US, OU=emSign PKI, O=eMudhra Inc, CN=emSign Root CA - C1
issuer=C=US, OU=emSign PKI, O=eMudhra Inc, CN=emSign Root CA - C1
subject=C=IN, OU=emSign PKI, O=eMudhra Technologies Limited, CN=emSign Root CA - G1
issuer=C=IN, OU=emSign PKI, O=eMudhra Technologies Limited, CN=emSign Root CA - G1
emSign ECC Root CA - G3, emSign Root TLS CA - G3, and AAA Certificate Services were absent. Python’s default SSL context reported 148 CA certificates and the same three emSign subjects. That is the patched Debian bundle as seen by a client, not a second copy of the certificate files.
NSS 3.131.1 removes emSign Root CA - C1, emSign ECC Root CA - C3, and emSign ECC Root CA - G3. The announcement ties the G3 removal to this incident and lists the other two removals under a different bug. Node.js 22.23.4 says its bundled roots were updated to that NSS release. Its tls.rootCertificates array contained 116 certificates. The only emSign subject in that array was emSign Root CA - G1. C1, C3, ECC G3, and AAA Certificate Services were not in it.
Copying the NSS removal list onto Debian would disable C1 and C3, which 20250419+deb13u1 left enabled. Copying the Debian blacklist onto Node would not explain Node’s store: Node had already dropped C1 and C3. The chain check below is the decision for the published intermediates. It is not a general opinion about the roots NSS removed for the other bug.
Verify the served chain
The incident site publishes chains.zip. Each file is the chain a named test host sent, leaf first. The two files that cover the trust paths are dv-g3b-emudhra.we-have-a-problem.com.pem and dv-g3b-aaa-emudhra.we-have-a-problem.com.pem. The first contains the leaf, EM DV TLS CA - G3B issued by emSign Root TLS CA - G3, and that root issued by emSign ECC Root CA - G3. The second contains the leaf and EM DV TLS CA - G3B issued by AAA Certificate Services. It does not carry the ECC cross-sign.
Split the leaf from the certificates the client would have to use as untrusted chain material. This awk program matched the split used for the verification below.
awk 'BEGIN{n=0} /-----BEGIN CERTIFICATE-----/{n++} n==1{print > "leaf.pem"} n>1{print > "untrusted.pem"}' \
dv-g3b-emudhra.we-have-a-problem.com.pem
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt \
-untrusted untrusted.pem leaf.pem
Against the patched bundle that command exited 2. OpenSSL reported error 20 at depth 2, on emSign Root TLS CA - G3: unable to get local issuer certificate. Depth 2 is the cross-signed root. The leaf and the intermediate were not the certificates OpenSSL was unable to place. The missing issuer was emSign ECC Root CA - G3, the root the Debian upload blacklisted.
The AAA file, split the same way, also exited 2. The error was at depth 1, on EM DV TLS CA - G3B, again unable to get local issuer certificate. Depth 1 is the intermediate. Its issuer, AAA Certificate Services, is the other root named in the changelog. The ECC G3 file does not satisfy that chain. Supplying it as -CAfile to the AAA leaf also exited 2.
The published intermediate inventory contains 16 certificates: eight intermediate names, each present once under an eMudhra issuer and once under the AAA cross-sign, including two code-signing names. openssl verify -purpose sslserver -CAfile /etc/ssl/certs/ca-certificates.crt failed all 16 with error 20. The code-signing certificates failed on the missing issuer before a TLS purpose check could accept them. That is not evidence they would have been valid server certificates. It is evidence that neither issuer path is anchored in this bundle.
A complete chain can still be distinguished from a corrupt download without putting the removed roots back into the system store. When the pre-update ECC G3 file was passed only as -CAfile, and the served untrusted certificates were passed with -untrusted, the G3 leaf verified and printed OK. The AAA leaf verified only when the pre-update AAA root was used the same way. Those two files are the ones the security update removed. Do not install the old package over 20250419+deb13u1, and do not append either root to ca-certificates.crt. The OK result is a control for the chain file. The deployment result is the error 20 against the active bundle.
Curl, Python, and Node do not fail the same way
Offline verification does not show what a client will print, because the clients do not share one trust implementation. The four public hosts used here were:
https://dv-g3b-emudhra.we-have-a-problem.com/
https://dv-g3b-aaa-emudhra.we-have-a-problem.com/
https://dv-g3b-noct-emudhra.we-have-a-problem.com/
https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
The noct hosts serve chains that were not logged to Certificate Transparency. The incident page says a successful connection to those hosts means the client still trusts a compromised intermediate and does not enforce CT. curl does not enforce CT. A certificate error from curl is still the result you want.
curl --silent --show-error --max-time 25 --output /dev/null \
--write-out 'http_code=%{http_code} ssl_verify=%{ssl_verify_result}\n' \
https://example.com/
curl --silent --show-error --max-time 25 --output /dev/null \
--write-out 'http_code=%{http_code} ssl_verify=%{ssl_verify_result}\n' \
https://dv-g3b-emudhra.we-have-a-problem.com/
curl --silent --show-error --max-time 25 --output /dev/null \
--write-out 'http_code=%{http_code} ssl_verify=%{ssl_verify_result}\n' \
https://dv-g3b-aaa-emudhra.we-have-a-problem.com/
example.com returned http_code=200 and ssl_verify=0. Both eMudhra hosts exited 60. curl printed SSL certificate problem: unable to get local issuer certificate, and the write-out line was http_code=000 ssl_verify=20. The two noct hosts failed the same way. Exit 60 with result 20 matches the offline error. It does not match a DNS failure, a timeout, or an HTTP error from a completed TLS handshake.
Python 3.13.5 used the system default context. urllib.request.urlopen reached example.com with status 200. Each of the four test hosts raised URLError with CERTIFICATE_VERIFY_FAILED and unable to get local issuer certificate. That interpreter is older than Python 3.13.17. The 3.13.17 release note says the security release upgrades bundled pip so that installer images pick up certifi 2026.10.19. It does not replace the CA file used by Debian’s ssl module. On this host the module followed /etc/ssl/certs/ca-certificates.crt, so the package update, not a newer interpreter, was what made Python reject the chains.
Node.js 22.23.4 did not print the issuer error.
node -e 'const https=require("https"); const url=process.argv[1]; https.get(url,{timeout:20000},res=>{console.log("status="+res.statusCode); res.resume()}).on("error",e=>{console.log("error_code="+(e.code||"")); process.exit(1)})' \
https://dv-g3b-emudhra.we-have-a-problem.com/
node --use-openssl-ca -e 'const https=require("https"); const url=process.argv[1]; https.get(url,{timeout:20000},res=>{console.log("status="+res.statusCode); res.resume()}).on("error",e=>{console.log("error_code="+(e.code||"")); process.exit(1)})' \
https://dv-g3b-emudhra.we-have-a-problem.com/
Both commands exited 1 with error_code=CERT_REVOKED. The same code came back for the AAA host and for both noct hosts. example.com returned status=200 in both modes. --use-openssl-ca selects the OpenSSL trust store instead of Node’s bundled roots. It did not make Node agree with curl. The 22.23.4 release notes include a change that rejects chains containing blocked public keys, and they record the root update to NSS 3.131.1. This run did not disconnect the host from the network, so it does not prove that Node skipped every revocation lookup. It does show that pointing Node at the system store was not enough to change CERT_REVOKED into the issuer error OpenSSL printed for the same chains.
Treat those as different observations. curl and Python failed because the patched bundle has no issuer for the served chain. Node 22.23.4 failed with a revoked-certificate error even when told to use that bundle. A deployment that checks only Node, or only curl, will record one of those facts and miss the other.
A certifi version does not name the file
Debian’s python3-certifi package on this host was 2025.1.31+ds-1. Importing it printed version 2025.01.31 and certifi.where() returned /etc/ssl/certs/ca-certificates.crt. The incident inventory lists certifi 2026.10.19 as the patched PyPI release. That version comparison would call this installation unpatched. The file it returned was the patched system bundle.
python3 -c 'import certifi; print(certifi.__version__); print(certifi.where())'
dpkg-query -W python3-certifi
If where() prints a path inside a virtual environment or a copy of cacert.pem, that file is the one to inspect, and the system blacklist does not cover it. If it prints the system bundle, the ca-certificates version is the version that matters. The PyPI release number is evidence about a different file until the path says otherwise.
The same split applies to other bundled stores. Node’s 116-certificate store and Debian’s 148-certificate context were both current for their own update paths, and they still did not contain the same emSign roots. An image that copies a CA bundle at build time will keep the old anchors until that image is rebuilt, even if the host package is already 20250419+deb13u1.
Do not remove every remaining emSign root
The first comment on Debian bug 1151413 named emSign ECC Root CA - C3, emSign ECC Root CA - G3, emSign Root CA - C1, and emSign Root CA - G1 as the affected certificates. The security upload did not blacklist that set. It blacklisted the ECC G3 root and the AAA root. C1, C3, and G1 were still enabled, still present as files, and still present in the active bundle after the update.
A later reader who applies the bug comment instead of the changelog will disable roots the security update left in place. A reader who greps for eMudhra, sees three hits, and concludes that the update was not installed will miss the two roots that are gone. The changelog, the ! lines, the missing files, and error 20 on both served chains are the four observations that agree. The leftover subjects do not contradict them.
NSS removed C1 and C3 in 3.131.1 under a separate bug. That is a reason to read that bug before treating Debian’s remaining roots as a permanent allow. It is not a reason to infer, from this incident’s intermediates, that those roots signed the compromised certificates. They did not appear in the served chains, and the intermediates did not verify against the ECC G3 certificate when it was offered as their issuer.
The result on this host is narrow and checkable. ca-certificates 20250419+deb13u1 removed the two trust anchors the served chains need. Three other emSign roots remained. curl and Python rejected the public test hosts with an issuer error, and a known site still verified. Node.js 22.23.4 rejected the same hosts with CERT_REVOKED, including when it was pointed at the system store. That is the difference a version string does not show.