Searching Certificate Transparency Logs (Part 2)

Abstract

In the last post we discussed why we’re building our own Certificate Transparency (CT) search tool. There’s good background on the CT ecosystem in that post, so check it out if you haven’t. This post assumes a certain understanding of terminology covered previously.

Now that we know where the CT logs live, and the different kinds of logs, we need to start reading them. This post will cover code examples to read log entries from both tiled and RFC 6962 logs, and discuss options for running at scale.

Golang is the language of Certificate Transparency

We’re C#/.NET guys by disposition (yeah, yeah) but we’re also pragmatists. While it’s possible to consume CT logs with any language, the guys writing the Web PKI plumbing are mostly writing in Golang (Go, from here on).

And if you’re willing to play in the Go sandbox, there are some nice client libraries available to consume both types of logs.

RFC 6962 Log Client

For all things RFC 6962 you should check out Google’s certificate-transparency-go library. In addition to containing a great client for interacting with log APIs, it contains some code to handle parallel scanning for you (more on that later).

Tiled Log Client

The tiled/static-ct logs are still pretty new, and we only found a single client implementation worth using. It’s written by the guy who proposed the tiled logs in the first place, so probably as good as it gets for now.

Two logs, two examples each

Let’s start calling the CT log APIs and see what we’re working with. Remember, there are two types of logs, and the read paths for each log are different. So for most things there will be two code examples - one for the RFC 6962 logs (the OGs) and one for the newer tiled logs.

RFC 6962: Getting the Signed Tree Head

RFC 6962 logs publish a Signed Tree Head (STH). This is a cryptographic snapshot of the Merkle tree root. Its purpose is to prove the tree hasn’t been tampered with, but for our puposes we primarily care about the tree_size value.

The tree_size is simply how large the log is currently. How many certificates and precertificates are in it. This is useful when scanning the logs - so if you stop or have to restart you know where you left off, and how far you are from completion.

STH In the Browser

You can actually get the signed tree head in the browser for most logs. For example, here’s the STH for Google’s Argon 2026h1 log:

https://ct.googleapis.com/logs/us1/argon2026h1/ct/v1/get-sth.

STH Using Go

But of course for any kind of real log scanning we’ll want to get it programmatically. Something like this will do the trick (but yours should have actual error handling):

import (
  // There are ambiguous "client" packages in the library so specify
  ctclient "github.com/google/certificate-transparency-go/client"
)

func main() {
  logUrl := "https://ct.googleapis.com/logs/us1/argon2026h1/"
  client, err := ctclient.New(logUrl, http DefaultClient, jsonclient.Options{})
  signedTreeHead, err := client.GetSTH(ctx)

  // Let's see how big the log is!
  fmt.Printf("Tree size: %d\n", signedTreeHead.TreeSize)
}

Tiled Logs: Getting the Checkpoint

Tiled logs don’t have a signed tree head. Instead they have a checkpoint. It’s not quite the same thing, but for our purposes it is - since it contains the log size!

Checkpoint in the Browser

Let’s Encrypt runs several tiled logs. As of November 2025 they are qualified for inclusion in Chrome, which means they’re ready for primetime. Let’s look at the checkpoint for Sycamore’s 2026h1 log:

https://mon.sycamore.ct.letsencrypt.org/2026h1/checkpoint

It’s also worth calling out that Sunlight logs have a little bit of UI to go with them. Here is Sycamore’s web UI: https://log.sycamore.ct.letsencrypt.org/

You can add items to the log right from your browser. There’s also helpful links to the checkpoint, and perhaps more importantly for later - the public key is also shown.

Checkpoint Using Go

We can get the checkpoint fairly easily in Go as well, but it’s a little more effort with the public key involved.

import (
	"filippo.io/sunlight"
)

func main() {
  // First up, we gotta turn the base64 key in to something usable.
  publicKey: "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEfEEe0JZknA91/c6eNl1aexgeKzuG
QUMvRCXPXg9L227O5I4Pi++Abcpq6qxlVUKPYafAJelAnMfGzv3lHCc8gA=="
  bytes, _ := base64.StdEncoding.DecodeString(publicKey)
  key, err := x509.ParsePKIXPublicKey(bytes)

  // Then we can make the client
  // for many of these logs the UserAgent is not optional.
  logMonitoringUrl :="https://mon.sycamore.ct.letsencrypt.org/2026h1/"
  client, err := sunlight.NewClient(&sunlight.ClientConfig{
    MonitoringPrefix: logMonitoringUrl,
    PublicKey:        key,
    UserAgent:        "YourUserAgent (you@yourdomain.com, +https://yourdomain.com)",
  })

  // Get the checkpoint (plumb in your context as available)
  checkpoint, _, _ := client.Checkpoint(context.TODO())

  fmt.Printf("Log size: %d\n", checkpoint.N)
}

RFC 6962: Scanning Entries

Now for the most important part: ripping through the log with reckless abandon, processing entries pell-mell. While the client in the Google CT library is useful, they’ve also helpfully included a scanner that will do the heavy lifting for you. It saves quite a bit of faffing.

Certificates vs. Precertificates

Before we see the code to scan through an RFC 6962 log, remember that there are two kinds of “things” in the logs. Precertificates and certificates. This distinction is important as we’ll see when scanning both kinds of logs.

Using the Scanner in Go

import (
  ctclient "github.com/google/certificate-transparency-go/client"
  "github.com/google/certificate-transparency-go/scanner"
)

func main () {
  logUrl := "https://ct.googleapis.com/logs/us1/argon2026h1/"
  client, err := ctclient.New(logUrl, http DefaultClient, jsonclient.Options{})
  s := scanner.NewScanner(client, scanner.DefaultScannerOptions())

  s.Scan(context.TODO(), func(raw *ct.RawLogEntry) {
		// this is the callback for certificates
		le, _ := raw.ToLogEntry()
		fmt.Printf("Cert common name: %s", le.X509Cert.Subject.CommonName)
	}, func(raw *ct.RawLogEntry) {
		// this is the callback for precertificates
		le, _ := raw.ToLogEntry()
		fmt.Printf("Precert common name: %s", le.Precert.TBSCertificate.Subject.CommonName)
	})
}

Processing at Scale

There are hundreds of millions (or sometimes billions) of items in large certificate transparency logs. Unfortunately, many operators of RFC 6962 logs throttle traffic heavily (looking at you Google). You can change the number of parallel threads fetching log entries in the scanner options, but in our experience you will get throttled very quickly at anything above 2 parallel threads.

The Cloudflare Nimbus logs were the most performant RFC 6962 logs we found. Google and Digicert both either throttled heavily, or had very slow to respond logs. Scanning tiled logs is an order of magnitude faster, so hopefully those become the standard (they’re a cost/performance win for log operators too!)

Tiled Logs: Scanning Tiles Like a Boss

The reason the read path differs for tile logs is because they’re optimized to be stored in object storage (think S3). This makes it easy for CDNs to front these logs and allows for blazing fast (relative to RFC 6962, anyways) log retrieval.

Let’s see how it works.

Scanning Tiled Logs in Go

import (
	"filippo.io/sunlight"
)

func main() {

  client := GetSunlightClient()
  checkpoint, _, _ := client.Checkpoint(context.TODO())

  for i, entry := range client.Entries(context.TODO(), checkpoint.Tree, 0) {
    if entry.IsPrecert {
      cert, err = x509.ParseCertificate(entry.PreCertificate)
      isPrecert = true
    } else {
      cert, err = x509.ParseCertificate(entry.Certificate)
    }
  }

  fmt.Printf("Log size: %d\n", checkpoint.N)
}

func GetSunlightClient() *sunlight.Client {
  // First up, we gotta turn the base64 key in to something usable.
  publicKey: "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEfEEe0JZknA91/c6eNl1aexgeKzuG
QUMvRCXPXg9L227O5I4Pi++Abcpq6qxlVUKPYafAJelAnMfGzv3lHCc8gA=="
  bytes, _ := base64.StdEncoding.DecodeString(publicKey)
  key, err := x509.ParsePKIXPublicKey(bytes)

  // Then we can make the client
  // for many of these logs the UserAgent is not optional.
  logMonitoringUrl :="https://mon.sycamore.ct.letsencrypt.org/2026h1/"
  client, err := sunlight.NewClient(&sunlight.ClientConfig{
    MonitoringPrefix: logMonitoringUrl,
    PublicKey:        key,
    UserAgent:        "YourUserAgent (you@yourdomain.com, +https://yourdomain.com)",
  })

  return client
}

The client.Entries() returns an iterator that we can just loop over to get the log entries. As with the RFC 6962 logs, there is a distinction between certificates and precertificates. The client itself does a decent job of multithreading.

Processing Tiled Logs at Scale

We were able to process over a hundred million records from each tiled log per day. This stands in stark contrast to the paltry low millions we could retrieve from the RFC boys.

There are a number of tiled log providers who are either usable or qualified in modern browsers, and Let’s Encrypt is going to switch to tiled logs exclusively at some point. From a scanning perspective they are clearly superior!

Conclusion

Using pre-built Go libraries, you can start scanning CT logs quickly. Though there are billions of certificates in the logs, if you parallelize your scanning, and run multiple logs at a time, you can fairly quickly scan a reasonably comprehensive chunk of the data. You can check out how it works using our free Certificate Transparency search tool.

In the next post we’ll talk about where to store it. Certainly we can’t enumerate every log every time there’s a search query!