diff options
| author | bndw <ben@bdw.to> | 2026-02-14 18:56:19 -0800 |
|---|---|---|
| committer | bndw <ben@bdw.to> | 2026-02-14 18:56:19 -0800 |
| commit | 7a5d5a53e5d6878f38382c4d35f644e088d318d2 (patch) | |
| tree | 5f8ba0bee800a5998ec4167c47e4adc6602243e1 /BENCHMARK_SUMMARY.md | |
| parent | 7fba76d7e4e63e0c29da81d6be43330743af1aaf (diff) | |
feat: add library comparison benchmarks with build tag isolation
Add comprehensive benchmarks comparing NWIO against nbd-wtf/go-nostr and
fiatjaf.com/nostr across event operations, signing, verification, and filtering.
Use build tag 'benchcmp' to prevent competitor libraries from polluting module
dependencies - they're only downloaded when explicitly running comparison tests.
Diffstat (limited to 'BENCHMARK_SUMMARY.md')
| -rw-r--r-- | BENCHMARK_SUMMARY.md | 154 |
1 files changed, 154 insertions, 0 deletions
diff --git a/BENCHMARK_SUMMARY.md b/BENCHMARK_SUMMARY.md new file mode 100644 index 0000000..e7f5b6d --- /dev/null +++ b/BENCHMARK_SUMMARY.md | |||
| @@ -0,0 +1,154 @@ | |||
| 1 | # Benchmark Results Summary | ||
| 2 | |||
| 3 | Comparison of three Go Nostr libraries: **NWIO** (northwest.io/nostr), **NBD** (github.com/nbd-wtf/go-nostr), and **Fiat** (fiatjaf.com/nostr) | ||
| 4 | |||
| 5 | ## Quick Performance Overview | ||
| 6 | |||
| 7 | ### 🏆 Winners by Category | ||
| 8 | |||
| 9 | | Operation | Winner | Performance | | ||
| 10 | |-----------|--------|-------------| | ||
| 11 | | **Event Unmarshal** | NWIO/Fiat | ~2.5 µs (tied) | | ||
| 12 | | **Event Marshal** | NWIO | 1.79 µs (fastest, least memory) | | ||
| 13 | | **Event Serialize** | NBD | 129 ns (3x faster than NWIO) | | ||
| 14 | | **Compute ID** | Fiat | 276 ns (2x faster than NWIO) | | ||
| 15 | | **Generate Key** | NBD | 470 ns (80x faster!) | | ||
| 16 | | **Event Sign** | NBD/Fiat | ~59 µs (2x faster than NWIO) | | ||
| 17 | | **Event Verify** | NWIO | 99.7 µs (slightly faster) | | ||
| 18 | | **Filter Match** | NWIO | 7.1 ns (2x faster than Fiat) | | ||
| 19 | | **Filter Complex** | NWIO | 30.9 ns (fastest) | | ||
| 20 | |||
| 21 | ## Detailed Results | ||
| 22 | |||
| 23 | ### Event Unmarshaling (JSON → Event) | ||
| 24 | ``` | ||
| 25 | NWIO: 2,541 ns/op 888 B/op 17 allocs/op ⭐ FASTEST, LOW MEMORY | ||
| 26 | NBD: 2,832 ns/op 944 B/op 13 allocs/op | ||
| 27 | Fiat: 2,545 ns/op 752 B/op 10 allocs/op ⭐ LEAST MEMORY | ||
| 28 | ``` | ||
| 29 | **Analysis**: All three are very competitive. NWIO and Fiat are effectively tied. Fiat uses least memory. | ||
| 30 | |||
| 31 | ### Event Marshaling (Event → JSON) | ||
| 32 | ``` | ||
| 33 | NWIO: 1,790 ns/op 1,010 B/op 3 allocs/op ⭐ FASTEST, LEAST ALLOCS | ||
| 34 | NBD: 1,819 ns/op 1,500 B/op 6 allocs/op | ||
| 35 | Fiat: 1,971 ns/op 2,254 B/op 13 allocs/op | ||
| 36 | ``` | ||
| 37 | **Analysis**: NWIO is fastest with minimal allocations. Significant memory advantage over competitors. | ||
| 38 | |||
| 39 | ### Event Serialization (for ID computation) | ||
| 40 | ``` | ||
| 41 | NWIO: 391 ns/op 360 B/op 7 allocs/op | ||
| 42 | NBD: 129 ns/op 208 B/op 2 allocs/op ⭐ FASTEST, 3x faster | ||
| 43 | Fiat: 161 ns/op 400 B/op 3 allocs/op | ||
| 44 | ``` | ||
| 45 | **Analysis**: NBD dominates here with optimized serialization. NWIO has room for improvement. | ||
| 46 | |||
| 47 | ### Event ID Computation | ||
| 48 | ``` | ||
| 49 | NWIO: 608 ns/op 488 B/op 9 allocs/op | ||
| 50 | NBD: 302 ns/op 336 B/op 4 allocs/op | ||
| 51 | Fiat: 276 ns/op 400 B/op 3 allocs/op ⭐ FASTEST | ||
| 52 | ``` | ||
| 53 | **Analysis**: NBD and Fiat are 2x faster. NWIO should optimize ID computation path. | ||
| 54 | |||
| 55 | ### Key Generation | ||
| 56 | ``` | ||
| 57 | NWIO: 37,689 ns/op 208 B/op 4 allocs/op | ||
| 58 | NBD: 470 ns/op 369 B/op 8 allocs/op ⭐ FASTEST, 80x faster! | ||
| 59 | Fiat: 25,375 ns/op 272 B/op 5 allocs/op | ||
| 60 | ``` | ||
| 61 | **Analysis**: ⚠️ NWIO is significantly slower. NBD appears to use a different key generation strategy. This is the biggest performance gap. | ||
| 62 | |||
| 63 | ### Event Signing | ||
| 64 | ``` | ||
| 65 | NWIO: 129,854 ns/op 2,363 B/op 42 allocs/op | ||
| 66 | NBD: 59,069 ns/op 2,112 B/op 35 allocs/op ⭐ TIED FASTEST | ||
| 67 | Fiat: 58,572 ns/op 1,760 B/op 29 allocs/op ⭐ LEAST MEMORY | ||
| 68 | ``` | ||
| 69 | **Analysis**: NBD and Fiat are 2x faster. NWIO has more allocations in signing path. | ||
| 70 | |||
| 71 | ### Event Verification | ||
| 72 | ``` | ||
| 73 | NWIO: 99,744 ns/op 953 B/op 19 allocs/op ⭐ FASTEST | ||
| 74 | NBD: 105,995 ns/op 624 B/op 11 allocs/op ⭐ LEAST MEMORY | ||
| 75 | Fiat: 103,744 ns/op 640 B/op 9 allocs/op | ||
| 76 | ``` | ||
| 77 | **Analysis**: NWIO is slightly faster (6% faster than others). Very competitive across all three. | ||
| 78 | |||
| 79 | ### Filter Matching (Simple) | ||
| 80 | ``` | ||
| 81 | NWIO: 7.1 ns/op 0 B/op 0 allocs/op ⭐ FASTEST, 2x faster | ||
| 82 | NBD: 10.8 ns/op 0 B/op 0 allocs/op | ||
| 83 | Fiat: 16.4 ns/op 0 B/op 0 allocs/op | ||
| 84 | ``` | ||
| 85 | **Analysis**: NWIO excels at filter matching! Zero allocations across all libraries. | ||
| 86 | |||
| 87 | ### Filter Matching (Complex with Tags) | ||
| 88 | ``` | ||
| 89 | NWIO: 30.9 ns/op 0 B/op 0 allocs/op ⭐ FASTEST | ||
| 90 | NBD: 33.4 ns/op 0 B/op 0 allocs/op | ||
| 91 | Fiat: 42.6 ns/op 0 B/op 0 allocs/op | ||
| 92 | ``` | ||
| 93 | **Analysis**: NWIO maintains lead in complex filters. Important for relay implementations. | ||
| 94 | |||
| 95 | ## Optimization Opportunities for NWIO | ||
| 96 | |||
| 97 | ### High Priority 🔴 | ||
| 98 | 1. **Key Generation** - 80x slower than NBD | ||
| 99 | - Current: 37.7 µs | ||
| 100 | - Target: ~500 ns (similar to NBD) | ||
| 101 | - Impact: Critical for client applications | ||
| 102 | |||
| 103 | 2. **Event Signing** - 2x slower than competitors | ||
| 104 | - Current: 130 µs | ||
| 105 | - Target: ~60 µs (match NBD/Fiat) | ||
| 106 | - Impact: High for client applications | ||
| 107 | |||
| 108 | ### Medium Priority 🟡 | ||
| 109 | 3. **Event Serialization** - 3x slower than NBD | ||
| 110 | - Current: 391 ns | ||
| 111 | - Target: ~130 ns (match NBD) | ||
| 112 | - Impact: Used in ID computation | ||
| 113 | |||
| 114 | 4. **ID Computation** - 2x slower than competitors | ||
| 115 | - Current: 608 ns | ||
| 116 | - Target: ~280 ns (match Fiat) | ||
| 117 | - Impact: Affects every event processing | ||
| 118 | |||
| 119 | ## Current Strengths of NWIO ✅ | ||
| 120 | |||
| 121 | 1. **Filter Matching** - 2x faster than Fiat, fastest overall | ||
| 122 | 2. **Event Marshaling** - Fastest with minimal allocations | ||
| 123 | 3. **Event Verification** - Slightly faster than competitors | ||
| 124 | 4. **Memory Efficiency** - Competitive or better in most operations | ||
| 125 | |||
| 126 | ## Recommendations | ||
| 127 | |||
| 128 | ### For Relay Implementations | ||
| 129 | - **NWIO excels**: Best filter matching performance | ||
| 130 | - All three are competitive for event parsing/verification | ||
| 131 | |||
| 132 | ### For Client Implementations | ||
| 133 | - **NBD/Fiat preferred**: Much faster key generation and signing | ||
| 134 | - NWIO needs optimization in crypto operations | ||
| 135 | |||
| 136 | ### Overall Assessment | ||
| 137 | - **NWIO**: Best for relay/filter-heavy workloads | ||
| 138 | - **NBD**: Most balanced, excellent crypto performance | ||
| 139 | - **Fiat**: Good all-around, lowest memory in some operations | ||
| 140 | |||
| 141 | ## Running Your Own Benchmarks | ||
| 142 | |||
| 143 | ```bash | ||
| 144 | # Run all benchmarks | ||
| 145 | ./run_benchmarks.sh | ||
| 146 | |||
| 147 | # Compare specific operations | ||
| 148 | go test -bench=BenchmarkEventSign -benchmem comparison_bench_test.go | ||
| 149 | go test -bench=BenchmarkFilterMatch -benchmem comparison_bench_test.go | ||
| 150 | |||
| 151 | # Statistical analysis with benchstat | ||
| 152 | go test -bench=. -count=10 comparison_bench_test.go > results.txt | ||
| 153 | benchstat results.txt | ||
| 154 | ``` | ||
