summaryrefslogtreecommitdiffstats
path: root/benchmarks
diff options
context:
space:
mode:
Diffstat (limited to 'benchmarks')
-rw-r--r--benchmarks/BENCHMARKS.md130
-rw-r--r--benchmarks/BENCHMARK_SUMMARY.md154
-rw-r--r--benchmarks/benchmark_results.txt32
-rwxr-xr-xbenchmarks/run_benchmarks.sh35
4 files changed, 351 insertions, 0 deletions
diff --git a/benchmarks/BENCHMARKS.md b/benchmarks/BENCHMARKS.md
new file mode 100644
index 0000000..1a671c1
--- /dev/null
+++ b/benchmarks/BENCHMARKS.md
@@ -0,0 +1,130 @@
1# Nostr Library Benchmarks
2
3This directory contains comprehensive benchmarks comparing three popular Go Nostr libraries:
4
5- **NWIO** (`code.northwest.io/nostr`) - This library
6- **NBD** (`github.com/nbd-wtf/go-nostr`) - Popular community library
7- **Fiat** (`fiatjaf.com/nostr`) - Original implementation by Fiatjaf
8
9## Benchmark Categories
10
11### Event Operations
12- **Unmarshal**: Parsing JSON into Event struct
13- **Marshal**: Serializing Event struct to JSON
14- **Serialize**: Canonical serialization for ID computation
15- **ComputeID**: Computing event ID hash
16- **Sign**: Signing events with private key
17- **Verify**: Verifying event signatures
18
19### Key Operations
20- **GenerateKey**: Generating new private keys
21
22### Filter Operations
23- **FilterMatch**: Simple filter matching (kind, author)
24- **FilterMatchComplex**: Complex filter matching (with tags, prefix matching)
25
26## Running Benchmarks
27
28**Important**: The comparison benchmarks are in a separate module (`benchmarks/comparison/`) to avoid polluting the main module dependencies with competitor libraries.
29
30### Quick Start
31
32Run all comparison benchmarks:
33```bash
34cd benchmarks/comparison
35go test -bench=. -benchmem -benchtime=1s
36```
37
38### Specific Benchmark Groups
39
40Event unmarshaling:
41```bash
42cd benchmarks/comparison
43go test -bench=BenchmarkEventUnmarshal -benchmem
44```
45
46Event signing:
47```bash
48cd benchmarks/comparison
49go test -bench=BenchmarkEventSign -benchmem
50```
51
52Event verification:
53```bash
54cd benchmarks/comparison
55go test -bench=BenchmarkEventVerify -benchmem
56```
57
58Filter matching:
59```bash
60cd benchmarks/comparison
61go test -bench=BenchmarkFilterMatch -benchmem
62```
63
64### Compare Single Library
65
66NWIO only:
67```bash
68cd benchmarks/comparison
69go test -bench='.*_NWIO' -benchmem
70```
71
72NBD only:
73```bash
74cd benchmarks/comparison
75go test -bench='.*_NBD' -benchmem
76```
77
78Fiat only:
79```bash
80cd benchmarks/comparison
81go test -bench='.*_Fiat' -benchmem
82```
83
84## Analyzing Results
85
86Use `benchstat` for statistical analysis:
87
88```bash
89# Install benchstat
90go install golang.org/x/perf/cmd/benchstat@latest
91
92# Run benchmarks multiple times and compare
93cd benchmarks/comparison
94go test -bench=. -benchmem -count=10 > results.txt
95benchstat results.txt
96```
97
98Compare two specific libraries:
99```bash
100cd benchmarks/comparison
101go test -bench='.*_NWIO' -benchmem -count=10 > nwio.txt
102go test -bench='.*_NBD' -benchmem -count=10 > nbd.txt
103benchstat nwio.txt nbd.txt
104```
105
106## Understanding the Output
107
108Example output:
109```
110BenchmarkEventSign_NWIO-24 50000 35421 ns/op 1024 B/op 12 allocs/op
111```
112
113- `50000`: Number of iterations
114- `35421 ns/op`: Nanoseconds per operation (lower is better)
115- `1024 B/op`: Bytes allocated per operation (lower is better)
116- `12 allocs/op`: Memory allocations per operation (lower is better)
117
118## Performance Tips
119
1201. **Event Unmarshaling**: Critical for relay implementations
1212. **Event Signing**: Important for client implementations
1223. **Event Verification**: Critical for all implementations
1234. **Filter Matching**: Important for relay implementations with many subscriptions
124
125## Notes
126
127- All benchmarks use realistic event data
128- Benchmarks run with default Go test timeout
129- Results may vary based on hardware and system load
130- Use `-benchtime=5s` for more stable results on noisy systems
diff --git a/benchmarks/BENCHMARK_SUMMARY.md b/benchmarks/BENCHMARK_SUMMARY.md
new file mode 100644
index 0000000..2e087a3
--- /dev/null
+++ b/benchmarks/BENCHMARK_SUMMARY.md
@@ -0,0 +1,154 @@
1# Benchmark Results Summary
2
3Comparison of three Go Nostr libraries: **NWIO** (code.northwest.io/nostr), **NBD** (github.com/nbd-wtf/go-nostr), and **Fiat** (fiatjaf.com/nostr)
4
5## Quick Performance Overview
6
7### 🏆 Winners by Category
8
9| Operation | Winner | Performance |
10|-----------|--------|-------------|
11| **Event Unmarshal** | NWIO/Fiat | ~2.5 µs (tied) |
12| **Event Marshal** | NWIO | 1.79 µs (fastest, least memory) |
13| **Event Serialize** | NBD | 129 ns (3x faster than NWIO) |
14| **Compute ID** | Fiat | 276 ns (2x faster than NWIO) |
15| **Generate Key** | NBD | 470 ns (80x faster!) |
16| **Event Sign** | NBD/Fiat | ~59 µs (2x faster than NWIO) |
17| **Event Verify** | NWIO | 99.7 µs (slightly faster) |
18| **Filter Match** | NWIO | 7.1 ns (2x faster than Fiat) |
19| **Filter Complex** | NWIO | 30.9 ns (fastest) |
20
21## Detailed Results
22
23### Event Unmarshaling (JSON → Event)
24```
25NWIO: 2,541 ns/op 888 B/op 17 allocs/op ⭐ FASTEST, LOW MEMORY
26NBD: 2,832 ns/op 944 B/op 13 allocs/op
27Fiat: 2,545 ns/op 752 B/op 10 allocs/op ⭐ LEAST MEMORY
28```
29**Analysis**: All three are very competitive. NWIO and Fiat are effectively tied. Fiat uses least memory.
30
31### Event Marshaling (Event → JSON)
32```
33NWIO: 1,790 ns/op 1,010 B/op 3 allocs/op ⭐ FASTEST, LEAST ALLOCS
34NBD: 1,819 ns/op 1,500 B/op 6 allocs/op
35Fiat: 1,971 ns/op 2,254 B/op 13 allocs/op
36```
37**Analysis**: NWIO is fastest with minimal allocations. Significant memory advantage over competitors.
38
39### Event Serialization (for ID computation)
40```
41NWIO: 391 ns/op 360 B/op 7 allocs/op
42NBD: 129 ns/op 208 B/op 2 allocs/op ⭐ FASTEST, 3x faster
43Fiat: 161 ns/op 400 B/op 3 allocs/op
44```
45**Analysis**: NBD dominates here with optimized serialization. NWIO has room for improvement.
46
47### Event ID Computation
48```
49NWIO: 608 ns/op 488 B/op 9 allocs/op
50NBD: 302 ns/op 336 B/op 4 allocs/op
51Fiat: 276 ns/op 400 B/op 3 allocs/op ⭐ FASTEST
52```
53**Analysis**: NBD and Fiat are 2x faster. NWIO should optimize ID computation path.
54
55### Key Generation
56```
57NWIO: 37,689 ns/op 208 B/op 4 allocs/op
58NBD: 470 ns/op 369 B/op 8 allocs/op ⭐ FASTEST, 80x faster!
59Fiat: 25,375 ns/op 272 B/op 5 allocs/op
60```
61**Analysis**: ⚠️ NWIO is significantly slower. NBD appears to use a different key generation strategy. This is the biggest performance gap.
62
63### Event Signing
64```
65NWIO: 129,854 ns/op 2,363 B/op 42 allocs/op
66NBD: 59,069 ns/op 2,112 B/op 35 allocs/op ⭐ TIED FASTEST
67Fiat: 58,572 ns/op 1,760 B/op 29 allocs/op ⭐ LEAST MEMORY
68```
69**Analysis**: NBD and Fiat are 2x faster. NWIO has more allocations in signing path.
70
71### Event Verification
72```
73NWIO: 99,744 ns/op 953 B/op 19 allocs/op ⭐ FASTEST
74NBD: 105,995 ns/op 624 B/op 11 allocs/op ⭐ LEAST MEMORY
75Fiat: 103,744 ns/op 640 B/op 9 allocs/op
76```
77**Analysis**: NWIO is slightly faster (6% faster than others). Very competitive across all three.
78
79### Filter Matching (Simple)
80```
81NWIO: 7.1 ns/op 0 B/op 0 allocs/op ⭐ FASTEST, 2x faster
82NBD: 10.8 ns/op 0 B/op 0 allocs/op
83Fiat: 16.4 ns/op 0 B/op 0 allocs/op
84```
85**Analysis**: NWIO excels at filter matching! Zero allocations across all libraries.
86
87### Filter Matching (Complex with Tags)
88```
89NWIO: 30.9 ns/op 0 B/op 0 allocs/op ⭐ FASTEST
90NBD: 33.4 ns/op 0 B/op 0 allocs/op
91Fiat: 42.6 ns/op 0 B/op 0 allocs/op
92```
93**Analysis**: NWIO maintains lead in complex filters. Important for relay implementations.
94
95## Optimization Opportunities for NWIO
96
97### High Priority 🔴
981. **Key Generation** - 80x slower than NBD
99 - Current: 37.7 µs
100 - Target: ~500 ns (similar to NBD)
101 - Impact: Critical for client applications
102
1032. **Event Signing** - 2x slower than competitors
104 - Current: 130 µs
105 - Target: ~60 µs (match NBD/Fiat)
106 - Impact: High for client applications
107
108### Medium Priority 🟡
1093. **Event Serialization** - 3x slower than NBD
110 - Current: 391 ns
111 - Target: ~130 ns (match NBD)
112 - Impact: Used in ID computation
113
1144. **ID Computation** - 2x slower than competitors
115 - Current: 608 ns
116 - Target: ~280 ns (match Fiat)
117 - Impact: Affects every event processing
118
119## Current Strengths of NWIO ✅
120
1211. **Filter Matching** - 2x faster than Fiat, fastest overall
1222. **Event Marshaling** - Fastest with minimal allocations
1233. **Event Verification** - Slightly faster than competitors
1244. **Memory Efficiency** - Competitive or better in most operations
125
126## Recommendations
127
128### For Relay Implementations
129- **NWIO excels**: Best filter matching performance
130- All three are competitive for event parsing/verification
131
132### For Client Implementations
133- **NBD/Fiat preferred**: Much faster key generation and signing
134- NWIO needs optimization in crypto operations
135
136### Overall Assessment
137- **NWIO**: Best for relay/filter-heavy workloads
138- **NBD**: Most balanced, excellent crypto performance
139- **Fiat**: Good all-around, lowest memory in some operations
140
141## Running Your Own Benchmarks
142
143```bash
144# Run all benchmarks
145./run_benchmarks.sh
146
147# Compare specific operations
148go test -bench=BenchmarkEventSign -benchmem comparison_bench_test.go
149go test -bench=BenchmarkFilterMatch -benchmem comparison_bench_test.go
150
151# Statistical analysis with benchstat
152go test -bench=. -count=10 comparison_bench_test.go > results.txt
153benchstat results.txt
154```
diff --git a/benchmarks/benchmark_results.txt b/benchmarks/benchmark_results.txt
new file mode 100644
index 0000000..c3976e6
--- /dev/null
+++ b/benchmarks/benchmark_results.txt
@@ -0,0 +1,32 @@
1goos: linux
2goarch: amd64
3cpu: AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M
4BenchmarkEventUnmarshal_NWIO-24 498826 2541 ns/op 888 B/op 17 allocs/op
5BenchmarkEventUnmarshal_NBD-24 423019 2832 ns/op 944 B/op 13 allocs/op
6BenchmarkEventUnmarshal_Fiat-24 430042 2545 ns/op 752 B/op 10 allocs/op
7BenchmarkEventMarshal_NWIO-24 613165 1790 ns/op 1010 B/op 3 allocs/op
8BenchmarkEventMarshal_NBD-24 620986 1819 ns/op 1500 B/op 6 allocs/op
9BenchmarkEventMarshal_Fiat-24 621964 1971 ns/op 2254 B/op 13 allocs/op
10BenchmarkEventSerialize_NWIO-24 3059661 391.0 ns/op 360 B/op 7 allocs/op
11BenchmarkEventSerialize_NBD-24 8824029 128.8 ns/op 208 B/op 2 allocs/op
12BenchmarkEventSerialize_Fiat-24 6533536 160.9 ns/op 400 B/op 3 allocs/op
13BenchmarkComputeID_NWIO-24 2108437 608.0 ns/op 488 B/op 9 allocs/op
14BenchmarkComputeID_NBD-24 4072243 302.2 ns/op 336 B/op 4 allocs/op
15BenchmarkComputeID_Fiat-24 4421660 275.9 ns/op 400 B/op 3 allocs/op
16BenchmarkGenerateKey_NWIO-24 31942 37689 ns/op 208 B/op 4 allocs/op
17BenchmarkGenerateKey_NBD-24 2489169 469.6 ns/op 369 B/op 8 allocs/op
18BenchmarkGenerateKey_Fiat-24 45475 25375 ns/op 272 B/op 5 allocs/op
19BenchmarkEventSign_NWIO-24 9072 129854 ns/op 2363 B/op 42 allocs/op
20BenchmarkEventSign_NBD-24 20325 59069 ns/op 2112 B/op 35 allocs/op
21BenchmarkEventSign_Fiat-24 20613 58572 ns/op 1760 B/op 29 allocs/op
22BenchmarkEventVerify_NWIO-24 12009 99744 ns/op 953 B/op 19 allocs/op
23BenchmarkEventVerify_NBD-24 10000 105995 ns/op 624 B/op 11 allocs/op
24BenchmarkEventVerify_Fiat-24 10000 103744 ns/op 640 B/op 9 allocs/op
25BenchmarkFilterMatch_NWIO-24 167376669 7.091 ns/op 0 B/op 0 allocs/op
26BenchmarkFilterMatch_NBD-24 100000000 10.82 ns/op 0 B/op 0 allocs/op
27BenchmarkFilterMatch_Fiat-24 71761591 16.40 ns/op 0 B/op 0 allocs/op
28BenchmarkFilterMatchComplex_NWIO-24 39214178 30.88 ns/op 0 B/op 0 allocs/op
29BenchmarkFilterMatchComplex_NBD-24 35580048 33.40 ns/op 0 B/op 0 allocs/op
30BenchmarkFilterMatchComplex_Fiat-24 28026481 42.64 ns/op 0 B/op 0 allocs/op
31PASS
32ok command-line-arguments 40.651s
diff --git a/benchmarks/run_benchmarks.sh b/benchmarks/run_benchmarks.sh
new file mode 100755
index 0000000..87dfd17
--- /dev/null
+++ b/benchmarks/run_benchmarks.sh
@@ -0,0 +1,35 @@
1#!/bin/bash
2set -e
3
4# Colors for output
5GREEN='\033[0;32m'
6BLUE='\033[0;34m'
7YELLOW='\033[1;33m'
8NC='\033[0m' # No Color
9
10echo -e "${BLUE}Running Nostr Library Benchmarks${NC}"
11echo -e "${BLUE}Comparing: NWIO vs NBD-WTF vs Fiatjaf${NC}"
12echo ""
13
14# First, install comparison dependencies if needed
15echo -e "${YELLOW}Ensuring comparison dependencies are available...${NC}"
16go get -tags=benchcmp -t ./...
17echo ""
18
19# Run all benchmarks with the benchcmp build tag
20echo -e "${GREEN}Running all benchmarks...${NC}"
21go test -tags=benchcmp -bench=. -benchmem -benchtime=1s -run=^$ | tee benchmark_results.txt
22
23echo ""
24echo -e "${GREEN}Results saved to benchmark_results.txt${NC}"
25echo ""
26echo -e "${BLUE}To run specific benchmark groups:${NC}"
27echo " go test -tags=benchcmp -bench=BenchmarkEventUnmarshal -benchmem"
28echo " go test -tags=benchcmp -bench=BenchmarkEventSign -benchmem"
29echo " go test -tags=benchcmp -bench=BenchmarkEventVerify -benchmem"
30echo " go test -tags=benchcmp -bench=BenchmarkFilterMatch -benchmem"
31echo ""
32echo -e "${BLUE}To compare specific libraries:${NC}"
33echo " go test -tags=benchcmp -bench='.*_NWIO' -benchmem"
34echo " go test -tags=benchcmp -bench='.*_NBD' -benchmem"
35echo " go test -tags=benchcmp -bench='.*_Fiat' -benchmem"
generated by cgit v1.2.3 (git 2.39.1) at 2026-02-17 02:34:45 +0000