From 7a5d5a53e5d6878f38382c4d35f644e088d318d2 Mon Sep 17 00:00:00 2001 From: bndw Date: Sat, 14 Feb 2026 18:56:19 -0800 Subject: feat: add library comparison benchmarks with build tag isolation Add comprehensive benchmarks comparing NWIO against nbd-wtf/go-nostr and fiatjaf.com/nostr across event operations, signing, verification, and filtering. Use build tag 'benchcmp' to prevent competitor libraries from polluting module dependencies - they're only downloaded when explicitly running comparison tests. --- BENCHMARK_SUMMARY.md | 154 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 BENCHMARK_SUMMARY.md (limited to 'BENCHMARK_SUMMARY.md') diff --git a/BENCHMARK_SUMMARY.md b/BENCHMARK_SUMMARY.md new file mode 100644 index 0000000..e7f5b6d --- /dev/null +++ b/BENCHMARK_SUMMARY.md @@ -0,0 +1,154 @@ +# Benchmark Results Summary + +Comparison of three Go Nostr libraries: **NWIO** (northwest.io/nostr), **NBD** (github.com/nbd-wtf/go-nostr), and **Fiat** (fiatjaf.com/nostr) + +## Quick Performance Overview + +### 🏆 Winners by Category + +| Operation | Winner | Performance | +|-----------|--------|-------------| +| **Event Unmarshal** | NWIO/Fiat | ~2.5 µs (tied) | +| **Event Marshal** | NWIO | 1.79 µs (fastest, least memory) | +| **Event Serialize** | NBD | 129 ns (3x faster than NWIO) | +| **Compute ID** | Fiat | 276 ns (2x faster than NWIO) | +| **Generate Key** | NBD | 470 ns (80x faster!) | +| **Event Sign** | NBD/Fiat | ~59 µs (2x faster than NWIO) | +| **Event Verify** | NWIO | 99.7 µs (slightly faster) | +| **Filter Match** | NWIO | 7.1 ns (2x faster than Fiat) | +| **Filter Complex** | NWIO | 30.9 ns (fastest) | + +## Detailed Results + +### Event Unmarshaling (JSON → Event) +``` +NWIO: 2,541 ns/op 888 B/op 17 allocs/op ⭐ FASTEST, LOW MEMORY +NBD: 2,832 ns/op 944 B/op 13 allocs/op +Fiat: 2,545 ns/op 752 B/op 10 allocs/op ⭐ LEAST MEMORY +``` +**Analysis**: All three are very competitive. NWIO and Fiat are effectively tied. Fiat uses least memory. + +### Event Marshaling (Event → JSON) +``` +NWIO: 1,790 ns/op 1,010 B/op 3 allocs/op ⭐ FASTEST, LEAST ALLOCS +NBD: 1,819 ns/op 1,500 B/op 6 allocs/op +Fiat: 1,971 ns/op 2,254 B/op 13 allocs/op +``` +**Analysis**: NWIO is fastest with minimal allocations. Significant memory advantage over competitors. + +### Event Serialization (for ID computation) +``` +NWIO: 391 ns/op 360 B/op 7 allocs/op +NBD: 129 ns/op 208 B/op 2 allocs/op ⭐ FASTEST, 3x faster +Fiat: 161 ns/op 400 B/op 3 allocs/op +``` +**Analysis**: NBD dominates here with optimized serialization. NWIO has room for improvement. + +### Event ID Computation +``` +NWIO: 608 ns/op 488 B/op 9 allocs/op +NBD: 302 ns/op 336 B/op 4 allocs/op +Fiat: 276 ns/op 400 B/op 3 allocs/op ⭐ FASTEST +``` +**Analysis**: NBD and Fiat are 2x faster. NWIO should optimize ID computation path. + +### Key Generation +``` +NWIO: 37,689 ns/op 208 B/op 4 allocs/op +NBD: 470 ns/op 369 B/op 8 allocs/op ⭐ FASTEST, 80x faster! +Fiat: 25,375 ns/op 272 B/op 5 allocs/op +``` +**Analysis**: ⚠️ NWIO is significantly slower. NBD appears to use a different key generation strategy. This is the biggest performance gap. + +### Event Signing +``` +NWIO: 129,854 ns/op 2,363 B/op 42 allocs/op +NBD: 59,069 ns/op 2,112 B/op 35 allocs/op ⭐ TIED FASTEST +Fiat: 58,572 ns/op 1,760 B/op 29 allocs/op ⭐ LEAST MEMORY +``` +**Analysis**: NBD and Fiat are 2x faster. NWIO has more allocations in signing path. + +### Event Verification +``` +NWIO: 99,744 ns/op 953 B/op 19 allocs/op ⭐ FASTEST +NBD: 105,995 ns/op 624 B/op 11 allocs/op ⭐ LEAST MEMORY +Fiat: 103,744 ns/op 640 B/op 9 allocs/op +``` +**Analysis**: NWIO is slightly faster (6% faster than others). Very competitive across all three. + +### Filter Matching (Simple) +``` +NWIO: 7.1 ns/op 0 B/op 0 allocs/op ⭐ FASTEST, 2x faster +NBD: 10.8 ns/op 0 B/op 0 allocs/op +Fiat: 16.4 ns/op 0 B/op 0 allocs/op +``` +**Analysis**: NWIO excels at filter matching! Zero allocations across all libraries. + +### Filter Matching (Complex with Tags) +``` +NWIO: 30.9 ns/op 0 B/op 0 allocs/op ⭐ FASTEST +NBD: 33.4 ns/op 0 B/op 0 allocs/op +Fiat: 42.6 ns/op 0 B/op 0 allocs/op +``` +**Analysis**: NWIO maintains lead in complex filters. Important for relay implementations. + +## Optimization Opportunities for NWIO + +### High Priority 🔴 +1. **Key Generation** - 80x slower than NBD + - Current: 37.7 µs + - Target: ~500 ns (similar to NBD) + - Impact: Critical for client applications + +2. **Event Signing** - 2x slower than competitors + - Current: 130 µs + - Target: ~60 µs (match NBD/Fiat) + - Impact: High for client applications + +### Medium Priority 🟡 +3. **Event Serialization** - 3x slower than NBD + - Current: 391 ns + - Target: ~130 ns (match NBD) + - Impact: Used in ID computation + +4. **ID Computation** - 2x slower than competitors + - Current: 608 ns + - Target: ~280 ns (match Fiat) + - Impact: Affects every event processing + +## Current Strengths of NWIO ✅ + +1. **Filter Matching** - 2x faster than Fiat, fastest overall +2. **Event Marshaling** - Fastest with minimal allocations +3. **Event Verification** - Slightly faster than competitors +4. **Memory Efficiency** - Competitive or better in most operations + +## Recommendations + +### For Relay Implementations +- **NWIO excels**: Best filter matching performance +- All three are competitive for event parsing/verification + +### For Client Implementations +- **NBD/Fiat preferred**: Much faster key generation and signing +- NWIO needs optimization in crypto operations + +### Overall Assessment +- **NWIO**: Best for relay/filter-heavy workloads +- **NBD**: Most balanced, excellent crypto performance +- **Fiat**: Good all-around, lowest memory in some operations + +## Running Your Own Benchmarks + +```bash +# Run all benchmarks +./run_benchmarks.sh + +# Compare specific operations +go test -bench=BenchmarkEventSign -benchmem comparison_bench_test.go +go test -bench=BenchmarkFilterMatch -benchmem comparison_bench_test.go + +# Statistical analysis with benchstat +go test -bench=. -count=10 comparison_bench_test.go > results.txt +benchstat results.txt +``` -- cgit v1.2.3